2020-06-03T18:31:01.484 INFO:root:teuthology version: 1.0.0-5aab241 2020-06-03T18:31:01.487 DEBUG:teuthology.run:Teuthology command: teuthology --lock --archive /home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016 --block --owner scheduled_yuvalif@teuthology --description rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} --verbose --name yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi -- /tmp/teuthology-worker.ujc6j3b1.tmp 2020-06-03T18:31:01.518 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-03T18:31:01.608 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016 branch: master description: rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} email: null first_in_suite: false job_id: '5116016' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: master ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 230b0bb1973ab49f8bd55921d5cad7dec0204c35 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 230b0bb1973ab49f8bd55921d5cad7dec0204c35 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: fix_issue_43536 sha1: 390027420862af043e1952c99341e4057b1eb4e0 owner: scheduled_yuvalif@teuthology priority: 10 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 230b0bb1973ab49f8bd55921d5cad7dec0204c35 sleep_before_teardown: 0 suite: rgw:multisite suite_branch: fix_issue_43536 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_fix_issue_43536/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 390027420862af043e1952c99341e4057b1eb4e0 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.8436 2020-06-03T18:31:01.609 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_fix_issue_43536/qa; will attempt to use it 2020-06-03T18:31:01.609 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_fix_issue_43536/qa/tasks 2020-06-03T18:31:01.610 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-03T18:31:01.612 INFO:teuthology.task.internal:Checking packages... 2020-06-03T18:31:01.635 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '230b0bb1973ab49f8bd55921d5cad7dec0204c35' 2020-06-03T18:31:01.635 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-03T18:31:01.636 INFO:teuthology.packaging:ref: None 2020-06-03T18:31:01.636 INFO:teuthology.packaging:tag: None 2020-06-03T18:31:01.636 INFO:teuthology.packaging:branch: master 2020-06-03T18:31:01.637 INFO:teuthology.packaging:sha1: 230b0bb1973ab49f8bd55921d5cad7dec0204c35 2020-06-03T18:31:01.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=master 2020-06-03T18:31:02.320 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-2208-g14c44071cf-1bionic 2020-06-03T18:31:02.320 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-03T18:31:02.332 INFO:teuthology.task.internal:no buildpackages task found 2020-06-03T18:31:02.332 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-03T18:31:02.348 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-03T18:31:02.350 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-03T18:31:02.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:31:12.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:31:22.439 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:31:32.455 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:31:42.472 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:31:52.482 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:32:02.500 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-03T18:32:12.520 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuvalif@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016'} 2020-06-03T18:32:12.551 DEBUG:teuthology.lock.ops:locked ubuntu@smithi112.front.sepia.ceph.com, ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:32:12.610 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffa40123b70>, signals=[15]) 2020-06-03T18:32:12.809 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of ubuntu 18.04 2020-06-03T18:32:12.813 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of ubuntu 18.04 2020-06-03T18:32:13.114 INFO:teuthology.orchestra.console:Power off smithi061 2020-06-03T18:32:13.116 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-03T18:32:13.140 INFO:teuthology.orchestra.console:Power off smithi112 2020-06-03T18:32:13.141 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-03T18:32:13.162 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:13.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:17.192 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:17.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:21.310 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:21.331 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:21.535 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2020-06-03T18:32:21.636 INFO:teuthology.orchestra.console:Power on smithi061 2020-06-03T18:32:21.637 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-03T18:32:21.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:25.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:25.671 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2020-06-03T18:32:25.693 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:25.775 INFO:teuthology.orchestra.console:Power on smithi112 2020-06-03T18:32:25.776 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-03T18:32:25.821 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:29.816 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:29.849 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:30.157 INFO:teuthology.orchestra.console:Power on for smithi061 completed 2020-06-03T18:32:30.259 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2020-06-03T18:32:33.966 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-03T18:32:34.180 INFO:teuthology.orchestra.console:Power on for smithi112 completed 2020-06-03T18:32:34.281 INFO:teuthology.provision.fog.smithi112:Waiting for deploy to finish 2020-06-03T18:35:31.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:35.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:41.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:44.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:50.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:53.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:35:59.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:02.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:08.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:11.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:17.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:20.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:26.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:29.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:36:35.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:07.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:07.060 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:07.171 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:13.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:13.087 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:13.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:19.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:19.106 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:19.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:19.210 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:25.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:25.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:25.223 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:26.151 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:31.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:31.239 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-03T18:37:32.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:32.506 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-03T18:37:32.509 INFO:teuthology.orchestra.run.smithi112:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-03T18:37:32.867 INFO:teuthology.provision.fog.smithi112:Node is ready 2020-06-03T18:37:32.868 INFO:teuthology.orchestra.run.smithi112:> hostname 2020-06-03T18:37:32.914 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112.front.sepia.ceph.com 2020-06-03T18:37:32.915 INFO:teuthology.orchestra.run.smithi112:> grep smithi112.front.sepia.ceph.com /etc/hosts 2020-06-03T18:37:32.963 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.15.112 smithi112.front.sepia.ceph.com smithi112 2020-06-03T18:37:32.963 INFO:teuthology.orchestra.run.smithi112:> sudo hostname smithi112 2020-06-03T18:37:33.026 INFO:teuthology.orchestra.run.smithi112:> sudo sed -i -e 's/smithi112.front.sepia.ceph.com/smithi112/g' /etc/hosts 2020-06-03T18:37:33.085 INFO:teuthology.orchestra.run.smithi112:> sudo sed -i -e 's/172.21.15.112/172.21.15.112/g' /etc/hosts 2020-06-03T18:37:33.142 INFO:teuthology.orchestra.run.smithi112:> sudo sed -i -e 's/smithi112.front.sepia.ceph.com/smithi112/g' /etc/hostname 2020-06-03T18:37:33.198 INFO:teuthology.orchestra.run.smithi112:> sudo hostname smithi112 2020-06-03T18:37:33.254 INFO:teuthology.provision.fog.smithi112:Deploy complete! 2020-06-03T18:37:33.255 DEBUG:teuthology.parallel:result is None 2020-06-03T18:37:37.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:37.333 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-03T18:37:37.334 INFO:teuthology.orchestra.run.smithi061:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-03T18:37:37.675 INFO:teuthology.provision.fog.smithi061:Node is ready 2020-06-03T18:37:37.676 INFO:teuthology.orchestra.run.smithi061:> hostname 2020-06-03T18:37:37.725 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com 2020-06-03T18:37:37.726 INFO:teuthology.orchestra.run.smithi061:> grep smithi061.front.sepia.ceph.com /etc/hosts 2020-06-03T18:37:37.773 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2020-06-03T18:37:37.774 INFO:teuthology.orchestra.run.smithi061:> sudo hostname smithi061 2020-06-03T18:37:37.913 INFO:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/smithi061.front.sepia.ceph.com/smithi061/g' /etc/hosts 2020-06-03T18:37:37.928 INFO:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/172.21.15.61/172.21.15.61/g' /etc/hosts 2020-06-03T18:37:37.986 INFO:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/smithi061.front.sepia.ceph.com/smithi061/g' /etc/hostname 2020-06-03T18:37:38.042 INFO:teuthology.orchestra.run.smithi061:> sudo hostname smithi061 2020-06-03T18:37:38.095 INFO:teuthology.provision.fog.smithi061:Deploy complete! 2020-06-03T18:37:38.096 DEBUG:teuthology.parallel:result is None 2020-06-03T18:37:38.230 INFO:teuthology.lock.ops:Checking smithi112.front.sepia.ceph.com 2020-06-03T18:37:38.231 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-03T18:37:38.282 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com 2020-06-03T18:37:38.283 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-03T18:37:38.307 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: set os type and version on server 2020-06-03T18:37:38.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:37:38.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:38.386 INFO:teuthology.orchestra.run.smithi112:> true 2020-06-03T18:37:38.630 INFO:teuthology.orchestra.run.smithi112:> uname -m 2020-06-03T18:37:38.678 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2020-06-03T18:37:38.679 INFO:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2020-06-03T18:37:38.727 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2020-06-03T18:37:38.728 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-03T18:37:38.728 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2020-06-03T18:37:38.728 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2020-06-03T18:37:38.729 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-03T18:37:38.729 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="18.04" 2020-06-03T18:37:38.729 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-03T18:37:38.729 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-03T18:37:38.730 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-03T18:37:38.730 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-03T18:37:38.730 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=bionic 2020-06-03T18:37:38.730 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=bionic 2020-06-03T18:37:38.732 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2020-06-03T18:37:38.753 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server 2020-06-03T18:37:38.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:37:38.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:38.831 INFO:teuthology.orchestra.run.smithi061:> true 2020-06-03T18:37:39.027 INFO:teuthology.orchestra.run.smithi061:> uname -m 2020-06-03T18:37:39.076 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2020-06-03T18:37:39.077 INFO:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2020-06-03T18:37:39.125 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Ubuntu" 2020-06-03T18:37:39.125 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-03T18:37:39.126 INFO:teuthology.orchestra.run.smithi061.stdout:ID=ubuntu 2020-06-03T18:37:39.126 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE=debian 2020-06-03T18:37:39.126 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-03T18:37:39.127 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="18.04" 2020-06-03T18:37:39.127 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-03T18:37:39.127 INFO:teuthology.orchestra.run.smithi061.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-03T18:37:39.127 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-03T18:37:39.128 INFO:teuthology.orchestra.run.smithi061.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-03T18:37:39.128 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_CODENAME=bionic 2020-06-03T18:37:39.128 INFO:teuthology.orchestra.run.smithi061.stdout:UBUNTU_CODENAME=bionic 2020-06-03T18:37:39.129 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2020-06-03T18:37:39.152 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-06-03T18:37:39.195 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiCUn2qnQjdIV5jntilANWO2UBbEruuFZKvHh00cy0SBueJd7dfgIIHf+B/h5JIO0ikBsLwdmTbIbbiZfXZx1lJvxHZWqe9yTUxej/ILchRWjVs30LngjD7lOwmoHrucD49q4sOcg8mDNLvyR+IQF8WvmGqeGEurxE34fX0KaEPlQgT1ivivdM3iXC71lELrZ+66UdcMUI766Buh3ZsiI3b7BwiaU14boJqHU4L67I5S0s+ibfGMTXAbqoCjfSHvBrcE+t2tx9D6In+E9C6cp16f1Avgi8uoZY5frwkY7D24EvOFIF4Xwu/VAcr30m83iiYgvjr9Y6Muii4pljWRsr smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/nBEuRLx53Ty/6d9ONoTJd876aJq75fLqp/5Y+tmafsmNhROmBI6pG9rUzCKDGOM2iueSI4SFXe4kV2VTqfftCY4vrWRdK4Xy0gZVyh+6pRaH5SbIDei5TAaN1M16VHvn/QJ4dYBxl1Wh4PpmlSFZVA37UVHaOHmHdVVUNdxteqrihmljgDL6K5FBsxncLzUzmVvnDcVKvcJnB0aq3n8DCzvswlf2fwQm6lJ7VO5XPxsa4n3PZeh/YIcAyyNZyADQUeBHrOfl2Zv2hFoQuODijEcyUTSMYkwvuZZ+2ILyGbpfmK1m6z91t62jgnU1PAg2OAtnRhbdM6ZHb+45DEKr 2020-06-03T18:37:39.195 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-03T18:37:39.256 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-06-03T18:37:39.289 INFO:teuthology.task.internal:Saving configuration 2020-06-03T18:37:39.324 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-06-03T18:37:39.376 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-06-03T18:37:39.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-03 18:32:12.528857', 'description': '/home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPbz8rXsJ7i3cKEHTfHAIyyuiIqjvbRsZyePPW8p1FfK', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi112.front.sepia.ceph.com'} 2020-06-03T18:37:39.407 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-03 18:32:12.528877', 'description': '/home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICF2agii54DeWC2jQdQzAZXNAU+HCVgm68ZtasNITxgH', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi061.front.sepia.ceph.com'} 2020-06-03T18:37:39.407 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-06-03T18:37:39.433 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-06-03T18:37:39.434 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-06-03T18:37:39.434 INFO:teuthology.run_tasks:Running task console_log... 2020-06-03T18:37:39.535 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffa400a82f0>, signals=[15]) 2020-06-03T18:37:39.537 INFO:teuthology.run_tasks:Running task internal.connect... 2020-06-03T18:37:39.544 INFO:teuthology.task.internal:Opening connections... 2020-06-03T18:37:39.544 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:37:39.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:39.624 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:37:39.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:37:39.699 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-06-03T18:37:39.710 INFO:teuthology.orchestra.run.smithi112:> uname -m 2020-06-03T18:37:39.813 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2020-06-03T18:37:39.814 INFO:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2020-06-03T18:37:39.862 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2020-06-03T18:37:39.862 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-03T18:37:39.862 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2020-06-03T18:37:39.863 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2020-06-03T18:37:39.863 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-03T18:37:39.863 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="18.04" 2020-06-03T18:37:39.863 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-03T18:37:39.863 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-03T18:37:39.864 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-03T18:37:39.864 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-03T18:37:39.864 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=bionic 2020-06-03T18:37:39.864 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=bionic 2020-06-03T18:37:39.865 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2020-06-03T18:37:39.886 INFO:teuthology.orchestra.run.smithi061:> uname -m 2020-06-03T18:37:39.892 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2020-06-03T18:37:39.893 INFO:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2020-06-03T18:37:39.937 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Ubuntu" 2020-06-03T18:37:39.937 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-03T18:37:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:ID=ubuntu 2020-06-03T18:37:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE=debian 2020-06-03T18:37:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-03T18:37:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="18.04" 2020-06-03T18:37:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-03T18:37:39.939 INFO:teuthology.orchestra.run.smithi061.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-03T18:37:39.939 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-03T18:37:39.939 INFO:teuthology.orchestra.run.smithi061.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-03T18:37:39.939 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_CODENAME=bionic 2020-06-03T18:37:39.939 INFO:teuthology.orchestra.run.smithi061.stdout:UBUNTU_CODENAME=bionic 2020-06-03T18:37:39.940 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2020-06-03T18:37:39.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-06-03T18:37:39.989 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-06-03T18:37:39.999 INFO:teuthology.task.internal:Checking for old test directory... 2020-06-03T18:37:40.000 INFO:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2020-06-03T18:37:40.002 INFO:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2020-06-03T18:37:40.008 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-06-03T18:37:40.018 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-06-03T18:37:40.019 INFO:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2020-06-03T18:37:40.051 INFO:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2020-06-03T18:37:40.063 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-06-03T18:37:40.105 INFO:teuthology.run_tasks:Running task kernel... 2020-06-03T18:37:40.126 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-06-03T18:37:40.127 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-03T18:37:40.128 INFO:teuthology.orchestra.run.smithi112:> uname -r 2020-06-03T18:37:40.134 INFO:teuthology.orchestra.run.smithi112.stdout:4.15.0-99-generic 2020-06-03T18:37:40.134 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.15.0-99-generic 2020-06-03T18:37:40.135 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2020-06-03T18:37:40.255 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get update 2020-06-03T18:37:40.520 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:37:40.520 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-06-03T18:37:40.580 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-06-03T18:37:40.860 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-06-03T18:37:40.935 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [727 kB] 2020-06-03T18:37:41.065 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [685 kB] 2020-06-03T18:37:41.284 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [474 kB] 2020-06-03T18:37:41.317 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [230 kB] 2020-06-03T18:37:41.333 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [45.7 kB] 2020-06-03T18:37:41.335 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [11.4 kB] 2020-06-03T18:37:41.336 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [669 kB] 2020-06-03T18:37:41.362 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [951 kB] 2020-06-03T18:37:41.409 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [623 kB] 2020-06-03T18:37:41.445 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [222 kB] 2020-06-03T18:37:41.451 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [7,596 B] 2020-06-03T18:37:41.452 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2020-06-03T18:37:41.452 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,824 B] 2020-06-03T18:37:41.455 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [324 kB] 2020-06-03T18:37:41.479 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [55.3 kB] 2020-06-03T18:37:41.484 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [13.8 kB] 2020-06-03T18:37:41.485 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,018 kB] 2020-06-03T18:37:41.539 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,077 kB] 2020-06-03T18:37:41.573 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [335 kB] 2020-06-03T18:37:41.587 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [8,720 B] 2020-06-03T18:37:41.588 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [15.7 kB] 2020-06-03T18:37:41.588 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,384 B] 2020-06-03T18:37:42.384 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 7,759 kB in 2s (4,017 kB/s) 2020-06-03T18:37:43.754 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:37:43.798 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:37:43.872 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:37:44.097 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:37:44.098 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:37:44.172 INFO:teuthology.orchestra.run.smithi112.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:37:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:37:44.174 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:37:44.251 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:37:44.482 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:37:44.483 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:37:44.641 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2020-06-03T18:37:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-03T18:37:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-4.15.0-101-generic 2020-06-03T18:37:44.642 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-4.15.0-101-generic linux-modules-extra-4.15.0-101-generic 2020-06-03T18:37:44.643 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2020-06-03T18:37:44.643 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-06-03T18:37:44.643 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2020-06-03T18:37:44.644 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2020-06-03T18:37:44.674 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2020-06-03T18:37:44.674 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-03T18:37:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-4.15.0-101-generic linux-modules-4.15.0-101-generic 2020-06-03T18:37:44.675 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-4.15.0-101-generic 2020-06-03T18:37:44.676 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2020-06-03T18:37:44.676 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2020-06-03T18:37:44.857 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:37:44.858 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 65.8 MB of archives. 2020-06-03T18:37:44.858 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 333 MB of additional disk space will be used. 2020-06-03T18:37:44.858 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-101-generic amd64 4.15.0-101.102 [13.0 MB] 2020-06-03T18:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-101-generic amd64 4.15.0-101.102 [8,005 kB] 2020-06-03T18:37:46.064 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-101-generic amd64 4.15.0-101.102 [32.8 MB] 2020-06-03T18:37:47.028 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.101.91 [1,864 B] 2020-06-03T18:37:47.028 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.101.91 [2,464 B] 2020-06-03T18:37:47.029 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101 all 4.15.0-101.102 [10.9 MB] 2020-06-03T18:37:47.327 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101-generic amd64 4.15.0-101.102 [1,100 kB] 2020-06-03T18:37:47.358 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.101.91 [2,424 B] 2020-06-03T18:37:47.796 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 65.8 MB in 3s (24.7 MB/s) 2020-06-03T18:37:47.962 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-4.15.0-101-generic. 2020-06-03T18:37:47.999 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101994 files and directories currently installed.) 2020-06-03T18:37:48.002 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:37:48.048 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:37:49.714 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-4.15.0-101-generic. 2020-06-03T18:37:49.727 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:37:49.826 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:37:50.326 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-101-generic. 2020-06-03T18:37:50.338 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:37:50.367 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:37:54.282 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:37:54.418 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:37:54.759 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:37:54.887 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:37:55.207 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-4.15.0-101. 2020-06-03T18:37:55.218 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-4.15.0-101_4.15.0-101.102_all.deb ... 2020-06-03T18:37:55.248 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-03T18:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-4.15.0-101-generic. 2020-06-03T18:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:02.210 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:38:02.324 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:38:02.693 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-03T18:38:02.845 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:02.971 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:03.556 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (4.15.0.101.91) ... 2020-06-03T18:38:03.665 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:04.108 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:38:04.109 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-101-generic 2020-06-03T18:38:04.242 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:04.727 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (4.15.0.101.91) ... 2020-06-03T18:38:04.853 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (4.15.0.101.91) ... 2020-06-03T18:38:04.979 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:05.164 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-06-03T18:38:05.165 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:38:20.052 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-06-03T18:38:20.053 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2020-06-03T18:38:20.082 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2020-06-03T18:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:38:20.424 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-03T18:38:20.774 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-03T18:38:22.184 INFO:teuthology.orchestra.run.smithi112.stdout:done 2020-06-03T18:38:22.624 INFO:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2020-06-03T18:38:22.645 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2020-06-03T18:38:22.645 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2020-06-03T18:38:22.645 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2020-06-03T18:38:22.646 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2020-06-03T18:38:22.646 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 16 2020-06-03T18:38:22.646 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:38:22.646 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2020-06-03T18:38:22.646 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 4.15.0.101.91 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:38:22.647 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2020-06-03T18:38:22.648 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-99-generic Expected: 4.15.0-101-generic 2020-06-03T18:38:22.648 INFO:teuthology.orchestra.run.smithi061:> uname -r 2020-06-03T18:38:22.655 INFO:teuthology.orchestra.run.smithi061.stdout:4.15.0-99-generic 2020-06-03T18:38:22.655 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.15.0-99-generic 2020-06-03T18:38:22.656 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2020-06-03T18:38:22.759 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get update 2020-06-03T18:38:22.891 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-06-03T18:38:23.013 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:38:23.096 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-06-03T18:38:23.149 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [727 kB] 2020-06-03T18:38:23.244 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [474 kB] 2020-06-03T18:38:23.252 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [230 kB] 2020-06-03T18:38:23.263 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [45.7 kB] 2020-06-03T18:38:23.263 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [11.4 kB] 2020-06-03T18:38:23.263 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [669 kB] 2020-06-03T18:38:23.271 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [623 kB] 2020-06-03T18:38:23.279 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [222 kB] 2020-06-03T18:38:23.282 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [7,596 B] 2020-06-03T18:38:23.282 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2020-06-03T18:38:23.283 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,824 B] 2020-06-03T18:38:23.370 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-06-03T18:38:23.495 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [951 kB] 2020-06-03T18:38:23.827 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [685 kB] 2020-06-03T18:38:23.864 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [324 kB] 2020-06-03T18:38:23.881 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [55.3 kB] 2020-06-03T18:38:23.909 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [13.8 kB] 2020-06-03T18:38:23.909 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,077 kB] 2020-06-03T18:38:23.961 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,018 kB] 2020-06-03T18:38:23.999 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [335 kB] 2020-06-03T18:38:24.007 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [8,720 B] 2020-06-03T18:38:24.008 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [15.7 kB] 2020-06-03T18:38:24.008 INFO:teuthology.orchestra.run.smithi061.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,384 B] 2020-06-03T18:38:24.530 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 7,759 kB in 2s (4,991 kB/s) 2020-06-03T18:38:25.939 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:38:25.986 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:38:26.066 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:38:26.324 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:38:26.325 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:38:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:38:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:38:26.414 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:38:26.499 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:38:26.759 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:38:26.760 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:38:26.943 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2020-06-03T18:38:26.943 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-03T18:38:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-generic linux-image-4.15.0-101-generic 2020-06-03T18:38:26.944 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-4.15.0-101-generic linux-modules-extra-4.15.0-101-generic 2020-06-03T18:38:26.945 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2020-06-03T18:38:26.945 INFO:teuthology.orchestra.run.smithi061.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-06-03T18:38:26.945 INFO:teuthology.orchestra.run.smithi061.stdout:Recommended packages: 2020-06-03T18:38:26.946 INFO:teuthology.orchestra.run.smithi061.stdout: thermald 2020-06-03T18:38:26.979 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2020-06-03T18:38:26.979 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-03T18:38:26.980 INFO:teuthology.orchestra.run.smithi061.stdout: linux-image-4.15.0-101-generic linux-modules-4.15.0-101-generic 2020-06-03T18:38:26.980 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-extra-4.15.0-101-generic 2020-06-03T18:38:26.981 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2020-06-03T18:38:26.982 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-generic linux-image-generic 2020-06-03T18:38:27.164 INFO:teuthology.orchestra.run.smithi061.stdout:3 upgraded, 5 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:38:27.164 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 65.8 MB of archives. 2020-06-03T18:38:27.165 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 333 MB of additional disk space will be used. 2020-06-03T18:38:27.165 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-101-generic amd64 4.15.0-101.102 [13.0 MB] 2020-06-03T18:38:28.223 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-101-generic amd64 4.15.0-101.102 [8,005 kB] 2020-06-03T18:38:28.419 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-101-generic amd64 4.15.0-101.102 [32.8 MB] 2020-06-03T18:38:29.383 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.101.91 [1,864 B] 2020-06-03T18:38:29.383 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.101.91 [2,464 B] 2020-06-03T18:38:29.383 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101 all 4.15.0-101.102 [10.9 MB] 2020-06-03T18:38:29.718 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101-generic amd64 4.15.0-101.102 [1,100 kB] 2020-06-03T18:38:29.728 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.101.91 [2,424 B] 2020-06-03T18:38:30.219 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 65.8 MB in 3s (24.1 MB/s) 2020-06-03T18:38:30.403 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-4.15.0-101-generic. 2020-06-03T18:38:30.448 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101994 files and directories currently installed.) 2020-06-03T18:38:30.449 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../0-linux-modules-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:38:30.481 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:32.097 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-image-4.15.0-101-generic. 2020-06-03T18:38:32.109 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../1-linux-image-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:38:32.210 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:32.698 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-101-generic. 2020-06-03T18:38:32.705 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:38:32.742 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:36.482 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../3-linux-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:38:36.609 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:38:36.923 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:38:37.170 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:38:37.490 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-4.15.0-101. 2020-06-03T18:38:37.502 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../5-linux-headers-4.15.0-101_4.15.0-101.102_all.deb ... 2020-06-03T18:38:37.532 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-03T18:38:42.756 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-4.15.0-101-generic. 2020-06-03T18:38:42.778 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../6-linux-headers-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-03T18:38:42.840 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:45.280 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.101.91_amd64.deb ... 2020-06-03T18:38:45.404 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-03T18:38:45.712 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-03T18:38:45.850 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:45.993 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:46.569 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-generic (4.15.0.101.91) ... 2020-06-03T18:38:46.695 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:47.152 INFO:teuthology.orchestra.run.smithi061.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:38:47.153 INFO:teuthology.orchestra.run.smithi061.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-101-generic 2020-06-03T18:38:47.272 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:47.774 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-generic (4.15.0.101.91) ... 2020-06-03T18:38:47.900 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-generic (4.15.0.101.91) ... 2020-06-03T18:38:48.026 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-03T18:38:48.218 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-06-03T18:38:48.219 INFO:teuthology.orchestra.run.smithi061.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:39:03.346 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-06-03T18:39:03.347 INFO:teuthology.orchestra.run.smithi061.stdout:Sourcing file `/etc/default/grub' 2020-06-03T18:39:03.366 INFO:teuthology.orchestra.run.smithi061.stdout:Generating grub configuration file ... 2020-06-03T18:39:03.694 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:39:03.736 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:39:04.045 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-03T18:39:04.075 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-03T18:39:05.508 INFO:teuthology.orchestra.run.smithi061.stdout:done 2020-06-03T18:39:05.914 INFO:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2020-06-03T18:39:05.938 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2020-06-03T18:39:05.939 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2020-06-03T18:39:05.939 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2020-06-03T18:39:05.939 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2020-06-03T18:39:05.939 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 16 2020-06-03T18:39:05.940 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:39:05.940 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2020-06-03T18:39:05.940 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2020-06-03T18:39:05.940 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 4.15.0.101.91 2020-06-03T18:39:05.940 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:39:05.941 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2020-06-03T18:39:05.941 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2020-06-03T18:39:05.941 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:39:05.941 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2020-06-03T18:39:05.942 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-99-generic Expected: 4.15.0-101-generic 2020-06-03T18:39:05.942 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-06-03T18:39:05.942 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-06-03T18:39:05.942 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-06-03T18:39:05.943 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-06-03T18:39:05.943 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2020-06-03T18:39:05.943 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2020-06-03T18:39:06.049 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get update 2020-06-03T18:39:06.300 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:39:06.311 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:39:06.371 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:39:06.451 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:39:08.128 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:39:08.175 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:39:08.252 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:39:08.495 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:39:08.496 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:39:08.571 INFO:teuthology.orchestra.run.smithi112.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:39:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:39:08.573 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:39:08.648 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:39:08.904 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:39:08.905 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:39:09.089 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-03T18:39:09.090 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:39:09.092 INFO:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2020-06-03T18:39:09.114 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2020-06-03T18:39:09.115 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2020-06-03T18:39:09.116 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2020-06-03T18:39:09.116 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2020-06-03T18:39:09.116 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 16 2020-06-03T18:39:09.117 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:39:09.117 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2020-06-03T18:39:09.117 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2020-06-03T18:39:09.117 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 4.15.0.101.91 2020-06-03T18:39:09.118 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:39:09.118 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2020-06-03T18:39:09.118 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2020-06-03T18:39:09.118 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:39:09.119 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2020-06-03T18:39:09.119 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:39:09.163 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.J3jpIb7h0c 2020-06-03T18:39:09.163 INFO:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.J3jpIb7h0c 2020-06-03T18:39:09.219 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.J3jpIb7h0c 2020-06-03T18:39:09.337 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.J3jpIb7h0c is 8KB 2020-06-03T18:39:09.386 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.J3jpIb7h0c 2020-06-03T18:39:09.393 INFO:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-06-03T18:39:09.449 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-06-03T18:39:09.545 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-101-generic 2020-06-03T18:39:09.546 INFO:teuthology.orchestra.run.smithi112:> sudo update-grub 2020-06-03T18:39:10.873 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2020-06-03T18:39:10.897 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2020-06-03T18:39:11.115 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:39:11.127 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:39:11.423 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-03T18:39:11.437 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-03T18:39:12.593 INFO:teuthology.orchestra.run.smithi112.stderr:done 2020-06-03T18:39:12.594 INFO:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2020-06-03T18:39:12.596 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-06-03T18:39:12.597 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2020-06-03T18:39:12.597 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2020-06-03T18:39:12.665 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get update 2020-06-03T18:39:12.922 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:39:12.923 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:39:12.990 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:39:13.070 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:39:14.792 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:39:14.839 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:39:14.964 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:39:15.206 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:39:15.207 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:39:15.294 INFO:teuthology.orchestra.run.smithi061.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:39:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:39:15.296 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:39:15.373 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:39:15.608 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:39:15.609 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:39:15.815 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-03T18:39:15.815 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:39:15.816 INFO:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2020-06-03T18:39:15.833 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2020-06-03T18:39:15.833 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2020-06-03T18:39:15.834 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2020-06-03T18:39:15.834 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2020-06-03T18:39:15.834 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 16 2020-06-03T18:39:15.834 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 4.15.0.101.91 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2020-06-03T18:39:15.835 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2020-06-03T18:39:15.836 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:39:15.836 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2020-06-03T18:39:15.836 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:39:15.881 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.O6PtTVUeQO 2020-06-03T18:39:15.882 INFO:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub/grub.cfg /tmp/tmp.O6PtTVUeQO 2020-06-03T18:39:15.946 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.O6PtTVUeQO 2020-06-03T18:39:16.060 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.O6PtTVUeQO is 8KB 2020-06-03T18:39:16.070 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.O6PtTVUeQO 2020-06-03T18:39:16.077 INFO:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-06-03T18:39:16.135 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-06-03T18:39:16.257 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-101-generic 2020-06-03T18:39:16.258 INFO:teuthology.orchestra.run.smithi061:> sudo update-grub 2020-06-03T18:39:17.673 INFO:teuthology.orchestra.run.smithi061.stderr:Sourcing file `/etc/default/grub' 2020-06-03T18:39:17.690 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2020-06-03T18:39:17.917 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-03T18:39:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-03T18:39:18.214 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-03T18:39:18.226 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-03T18:39:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:done 2020-06-03T18:39:19.312 INFO:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2020-06-03T18:39:49.344 INFO:teuthology.misc:Re-opening connections... 2020-06-03T18:39:49.346 INFO:teuthology.misc:trying to connect to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:39:49.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:39:49.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:07.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2020-06-03T18:40:07.904 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:07.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:07.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:26.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:26.336 DEBUG:teuthology.misc:waited 36.98981809616089 2020-06-03T18:40:27.339 INFO:teuthology.misc:trying to connect to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:40:27.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:27.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:45.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2020-06-03T18:40:45.791 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:45.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:45.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:48.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:48.863 DEBUG:teuthology.misc:waited 59.5174605846405 2020-06-03T18:40:49.864 INFO:teuthology.misc:trying to connect to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:40:49.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:49.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:49.867 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2020-06-03T18:40:49.868 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:49.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:49.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:51.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:51.935 DEBUG:teuthology.misc:waited 62.589539766311646 2020-06-03T18:40:52.936 INFO:teuthology.misc:trying to connect to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:40:52.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:52.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:52.939 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2020-06-03T18:40:52.940 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:52.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:52.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:55.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:55.011 DEBUG:teuthology.misc:waited 65.6658124923706 2020-06-03T18:40:56.012 INFO:teuthology.misc:trying to connect to ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:40:56.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:56.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:56.124 INFO:teuthology.orchestra.run.smithi112:> true 2020-06-03T18:40:56.434 DEBUG:teuthology.misc:waited 67.08831191062927 2020-06-03T18:40:57.435 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:57.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:57.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:57.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:57.440 DEBUG:teuthology.misc:waited 68.09447121620178 2020-06-03T18:40:58.446 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:40:58.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:40:58.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:40:59.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:40:59.487 DEBUG:teuthology.misc:waited 70.14159774780273 2020-06-03T18:41:00.488 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:41:00.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:41:00.489 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:41:00.491 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:41:00.492 DEBUG:teuthology.misc:waited 71.14612007141113 2020-06-03T18:41:01.496 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:41:01.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:41:01.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:41:01.504 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:41:01.505 DEBUG:teuthology.misc:waited 72.15897369384766 2020-06-03T18:41:02.505 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:41:02.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:41:02.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:41:02.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:41:02.508 DEBUG:teuthology.misc:waited 73.16272449493408 2020-06-03T18:41:03.510 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:41:03.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:41:03.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:41:03.513 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2020-06-03T18:41:03.514 DEBUG:teuthology.misc:waited 74.16815209388733 2020-06-03T18:41:04.515 INFO:teuthology.misc:trying to connect to ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:41:04.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:41:04.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:41:04.719 INFO:teuthology.orchestra.run.smithi061:> true 2020-06-03T18:41:05.946 DEBUG:teuthology.misc:waited 76.60057592391968 2020-06-03T18:41:06.948 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-06-03T18:41:06.948 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-101-generic"... 2020-06-03T18:41:06.949 INFO:teuthology.orchestra.run.smithi112:> uname -r 2020-06-03T18:41:06.956 INFO:teuthology.orchestra.run.smithi112.stdout:4.15.0-101-generic 2020-06-03T18:41:06.956 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-101-generic vs 4.15.0-101-generic 2020-06-03T18:41:06.956 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-06-03T18:41:06.957 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-06-03T18:41:06.957 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-101-generic"... 2020-06-03T18:41:06.957 INFO:teuthology.orchestra.run.smithi061:> uname -r 2020-06-03T18:41:06.963 INFO:teuthology.orchestra.run.smithi061.stdout:4.15.0-101-generic 2020-06-03T18:41:06.964 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-101-generic vs 4.15.0-101-generic 2020-06-03T18:41:06.964 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-06-03T18:41:07.965 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-06-03T18:41:07.966 INFO:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-03T18:41:08.060 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2020-06-03T18:41:08.078 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-06-03T18:41:08.079 INFO:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-03T18:41:08.131 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2020-06-03T18:41:08.168 INFO:teuthology.run_tasks:Running task internal.base... 2020-06-03T18:41:08.178 INFO:teuthology.task.internal:Creating test directory... 2020-06-03T18:41:08.179 INFO:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-03T18:41:08.182 INFO:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-03T18:41:08.188 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-06-03T18:41:08.203 INFO:teuthology.run_tasks:Running task internal.archive... 2020-06-03T18:41:08.217 INFO:teuthology.task.internal:Creating archive directory... 2020-06-03T18:41:08.218 INFO:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-03T18:41:08.231 INFO:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-03T18:41:08.261 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-06-03T18:41:08.272 INFO:teuthology.task.internal:Enabling coredump saving... 2020-06-03T18:41:08.273 INFO:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-03T18:41:08.287 INFO:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-03T18:41:08.301 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-03T18:41:08.315 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-03T18:41:08.330 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-03T18:41:08.337 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-03T18:41:08.339 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-06-03T18:41:08.350 INFO:teuthology.task.internal:Configuring sudo... 2020-06-03T18:41:08.351 INFO:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-03T18:41:08.363 INFO:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-03T18:41:08.394 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-06-03T18:41:08.408 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-06-03T18:41:08.409 INFO:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-03T18:41:08.419 INFO:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-03T18:41:08.441 INFO:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-03T18:41:08.494 INFO:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-03T18:41:08.542 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-03T18:41:08.630 INFO:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-03T18:41:08.637 INFO:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-03T18:41:08.688 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-03T18:41:08.776 INFO:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2020-06-03T18:41:08.819 INFO:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2020-06-03T18:41:08.878 INFO:teuthology.run_tasks:Running task internal.timer... 2020-06-03T18:41:08.894 INFO:teuthology.task.internal:Starting timer... 2020-06-03T18:41:08.894 INFO:teuthology.run_tasks:Running task kernel... 2020-06-03T18:41:08.904 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-06-03T18:41:08.905 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-03T18:41:08.905 INFO:teuthology.orchestra.run.smithi112:> uname -r 2020-06-03T18:41:08.912 INFO:teuthology.orchestra.run.smithi112.stdout:4.15.0-101-generic 2020-06-03T18:41:08.912 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.15.0-101-generic 2020-06-03T18:41:08.912 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2020-06-03T18:41:09.026 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get update 2020-06-03T18:41:09.172 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:41:09.284 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:41:09.365 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:41:09.446 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:41:11.391 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:41:11.435 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:41:11.525 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:41:11.756 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:41:11.756 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:41:11.831 INFO:teuthology.orchestra.run.smithi112.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:41:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:41:11.833 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:41:11.912 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:41:12.159 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:41:12.160 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:41:12.346 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-03T18:41:12.346 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:41:12.347 INFO:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2020-06-03T18:41:12.368 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2020-06-03T18:41:12.368 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2020-06-03T18:41:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2020-06-03T18:41:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2020-06-03T18:41:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 16 2020-06-03T18:41:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:41:12.369 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2020-06-03T18:41:12.370 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2020-06-03T18:41:12.370 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 4.15.0.101.91 2020-06-03T18:41:12.370 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:41:12.370 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2020-06-03T18:41:12.371 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2020-06-03T18:41:12.371 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:41:12.371 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2020-06-03T18:41:12.371 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-03T18:41:12.372 INFO:teuthology.orchestra.run.smithi061:> uname -r 2020-06-03T18:41:12.378 INFO:teuthology.orchestra.run.smithi061.stdout:4.15.0-101-generic 2020-06-03T18:41:12.379 INFO:teuthology.task.kernel:Running kernel on smithi061: 4.15.0-101-generic 2020-06-03T18:41:12.379 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2020-06-03T18:41:12.485 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get update 2020-06-03T18:41:12.677 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:41:12.787 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:41:12.867 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:41:12.948 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:41:15.013 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:41:15.059 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-03T18:41:15.145 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:41:15.381 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:41:15.382 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:41:15.469 INFO:teuthology.orchestra.run.smithi061.stderr:E: Unable to locate package linux-image-current-generic 2020-06-03T18:41:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-03T18:41:15.471 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-03T18:41:15.554 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:41:15.814 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:41:15.815 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:41:16.022 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-03T18:41:16.023 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-03T18:41:16.024 INFO:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2020-06-03T18:41:16.047 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2020-06-03T18:41:16.047 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2020-06-03T18:41:16.048 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2020-06-03T18:41:16.048 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2020-06-03T18:41:16.048 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 16 2020-06-03T18:41:16.048 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2020-06-03T18:41:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2020-06-03T18:41:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2020-06-03T18:41:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 4.15.0.101.91 2020-06-03T18:41:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-03T18:41:16.050 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2020-06-03T18:41:16.050 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2020-06-03T18:41:16.050 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2020-06-03T18:41:16.050 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2020-06-03T18:41:16.051 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-03T18:41:16.051 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-06-03T18:41:16.051 INFO:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-03T18:41:16.065 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2020-06-03T18:41:16.170 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-06-03T18:41:16.170 INFO:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-03T18:41:16.184 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2020-06-03T18:41:16.292 INFO:teuthology.run_tasks:Running task pcp... 2020-06-03T18:41:16.306 INFO:teuthology.run_tasks:Running task selinux... 2020-06-03T18:41:16.335 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2020-06-03T18:41:16.336 DEBUG:teuthology.task.selinux:Excluding smithi061: OS 'ubuntu' does not support SELinux 2020-06-03T18:41:16.336 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-06-03T18:41:16.336 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-06-03T18:41:16.337 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-06-03T18:41:16.337 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-06-03T18:41:16.350 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-06-03T18:41:16.360 INFO:teuthology.repo_utils:Fetching master from origin 2020-06-03T18:41:16.424 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-06-03T18:41:16.445 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-06-03T18:41:16.448 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi061.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-06-03T18:41:16.965 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-06-03T18:41:18.496 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-06-03T18:41:21.113 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-06-03T18:41:21.136 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:21 +0000 (0:00:00.102) 0:00:00.102 ******** 2020-06-03T18:41:21.154 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:21 +0000 (0:00:00.017) 0:00:00.119 ******** 2020-06-03T18:41:22.962 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-06-03T18:41:22.963 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] 2020-06-03T18:41:23.036 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:23 +0000 (0:00:01.882) 0:00:02.001 ******** 2020-06-03T18:41:23.048 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] 2020-06-03T18:41:23.122 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:23 +0000 (0:00:00.085) 0:00:02.087 ******** 2020-06-03T18:41:23.485 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-06-03T18:41:23.486 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-03T18:41:23.521 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:23 +0000 (0:00:00.399) 0:00:02.487 ******** 2020-06-03T18:41:23.530 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-03T18:41:23.569 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:23 +0000 (0:00:00.047) 0:00:02.534 ******** 2020-06-03T18:41:24.185 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi112.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-06-03T18:41:24.223 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:24 +0000 (0:00:00.653) 0:00:03.188 ******** 2020-06-03T18:41:24.232 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-06-03T18:41:24.287 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:24 +0000 (0:00:00.064) 0:00:03.252 ******** 2020-06-03T18:41:24.710 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-06-03T18:41:24.712 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.062887", "end": "2020-06-03 18:41:24.370054", "rc": 0, "start": "2020-06-03 18:41:24.307167", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.071600", "end": "2020-06-03 18:41:24.277156", "rc": 0, "start": "2020-06-03 18:41:24.205556", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-06-03T18:41:24.760 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:24 +0000 (0:00:00.473) 0:00:03.725 ******** 2020-06-03T18:41:24.798 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:24 +0000 (0:00:00.038) 0:00:03.764 ******** 2020-06-03T18:41:25.135 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-06-03T18:41:25.137 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:41:25.172 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:25 +0000 (0:00:00.374) 0:00:04.138 ******** 2020-06-03T18:41:25.210 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:25 +0000 (0:00:00.037) 0:00:04.175 ******** 2020-06-03T18:41:26.154 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-06-03T18:41:26.155 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591209685.2274387-269294263226205/source", "state": "file", "uid": 0} changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591209685.1888175-243723885023207/source", "state": "file", "uid": 0} 2020-06-03T18:41:26.190 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:26 +0000 (0:00:00.979) 0:00:05.155 ******** 2020-06-03T18:41:26.230 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:26 +0000 (0:00:00.039) 0:00:05.195 ******** 2020-06-03T18:41:26.640 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-06-03T18:41:26.642 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-06-03T18:41:26.644 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2020-06-03T18:41:26.646 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2020-06-03T18:41:26.647 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-06-03T18:41:26.801 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-06-03T18:41:26.808 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-06-03T18:41:26.969 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:26 +0000 (0:00:00.739) 0:00:05.935 ******** 2020-06-03T18:41:27.004 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:26 +0000 (0:00:00.034) 0:00:05.969 ******** 2020-06-03T18:41:28.000 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi061.front.sepia.ceph.com] 2020-06-03T18:41:28.087 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-06-03T18:41:28.089 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:28 +0000 (0:00:01.083) 0:00:07.052 ******** 2020-06-03T18:41:28.524 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-06-03T18:41:28.526 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.185086", "end": "2020-06-03 18:41:28.485083", "rc": 0, "start": "2020-06-03 18:41:28.299997", "stderr": "", "stderr_lines": [], "stdout": "36acb243999edacffcb85d4e38b209d6e3ce3636", "stdout_lines": ["36acb243999edacffcb85d4e38b209d6e3ce3636"]} 2020-06-03T18:41:28.569 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:28 +0000 (0:00:00.482) 0:00:07.535 ******** 2020-06-03T18:41:28.724 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-06-03T18:41:28.728 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011231", "end": "2020-06-03 18:41:28.393106", "failed_when_result": false, "rc": 0, "start": "2020-06-03 18:41:28.381875", "stderr": "", "stderr_lines": [], "stdout": "f148133fb0c07a635fda93cc17e7dcc2a5d008b7", "stdout_lines": ["f148133fb0c07a635fda93cc17e7dcc2a5d008b7"]} 2020-06-03T18:41:28.809 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:28 +0000 (0:00:00.239) 0:00:07.774 ******** 2020-06-03T18:41:28.849 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-06-03T18:41:28.851 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:28.889 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:28 +0000 (0:00:00.079) 0:00:07.854 ******** 2020-06-03T18:41:28.911 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi112.front.sepia.ceph.com] 2020-06-03T18:41:28.955 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-06-03T18:41:28.957 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:28.989 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:28 +0000 (0:00:00.099) 0:00:07.954 ******** 2020-06-03T18:41:29.030 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.040) 0:00:07.994 ******** 2020-06-03T18:41:29.038 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-06-03T18:41:29.040 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.078 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.048) 0:00:08.043 ******** 2020-06-03T18:41:29.108 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-06-03T18:41:29.110 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.146 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.068) 0:00:08.112 ******** 2020-06-03T18:41:29.186 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-06-03T18:41:29.188 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.222 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.075) 0:00:08.188 ******** 2020-06-03T18:41:29.369 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-06-03T18:41:29.371 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:41:29.406 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.183) 0:00:08.371 ******** 2020-06-03T18:41:29.447 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-06-03T18:41:29.449 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.193048", "end": "2020-06-03 18:41:29.415316", "rc": 0, "start": "2020-06-03 18:41:29.222268", "stderr": "", "stderr_lines": [], "stdout": "36acb243999edacffcb85d4e38b209d6e3ce3636", "stdout_lines": ["36acb243999edacffcb85d4e38b209d6e3ce3636"]} 2020-06-03T18:41:29.484 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.077) 0:00:08.449 ******** 2020-06-03T18:41:29.605 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:29.607 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-03T18:41:29.648 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017243", "end": "2020-06-03 18:41:29.216788", "failed_when_result": false, "rc": 0, "start": "2020-06-03 18:41:29.199545", "stderr": "", "stderr_lines": [], "stdout": "f148133fb0c07a635fda93cc17e7dcc2a5d008b7", "stdout_lines": ["f148133fb0c07a635fda93cc17e7dcc2a5d008b7"]} 2020-06-03T18:41:29.687 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.203) 0:00:08.653 ******** 2020-06-03T18:41:29.727 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-06-03T18:41:29.728 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.764 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.076) 0:00:08.729 ******** 2020-06-03T18:41:29.773 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:29.774 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-03T18:41:29.796 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.832 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.067) 0:00:08.797 ******** 2020-06-03T18:41:29.872 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-06-03T18:41:29.874 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.919 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.087) 0:00:08.884 ******** 2020-06-03T18:41:29.929 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:29.931 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-03T18:41:29.952 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-06-03T18:41:29.954 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:29.988 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:29 +0000 (0:00:00.069) 0:00:08.954 ******** 2020-06-03T18:41:30.017 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:30.019 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:30.054 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:30 +0000 (0:00:00.066) 0:00:09.020 ******** 2020-06-03T18:41:30.163 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:30.166 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-03T18:41:30.206 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-06-03T18:41:30.208 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:41:30.243 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:30 +0000 (0:00:00.188) 0:00:09.208 ******** 2020-06-03T18:41:30.296 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:30.298 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-03T18:41:30.437 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-03T18:41:30.438 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-03T18:41:30.574 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-03T18:41:30.575 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-03T18:41:30.709 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-03T18:41:30.711 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-03T18:41:30.846 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-03T18:41:30.847 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-06-03T18:41:30.981 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-03T18:41:30.983 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-03T18:41:31.109 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-03T18:41:31.120 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-03T18:41:31.245 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-03T18:41:31.308 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-03T18:41:31.384 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-03T18:41:31.468 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-03T18:41:31.542 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-03T18:41:31.606 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-03T18:41:31.676 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-03T18:41:31.742 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-03T18:41:31.816 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-03T18:41:31.879 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-03T18:41:31.954 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-03T18:41:32.007 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-03T18:41:32.101 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-03T18:41:32.150 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-03T18:41:32.235 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-03T18:41:32.278 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-03T18:41:32.375 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-03T18:41:32.418 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-03T18:41:32.522 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-03T18:41:32.578 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-03T18:41:32.660 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-03T18:41:32.703 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-03T18:41:32.805 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-03T18:41:32.838 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-03T18:41:32.943 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-03T18:41:32.991 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-03T18:41:33.085 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-03T18:41:33.127 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-03T18:41:33.226 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-03T18:41:33.258 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-03T18:41:33.363 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-03T18:41:33.409 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-03T18:41:33.513 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-06-03T18:41:33.555 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-03T18:41:33.643 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-03T18:41:33.685 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-03T18:41:33.788 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-03T18:41:33.810 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-03T18:41:33.930 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-03T18:41:33.952 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-03T18:41:34.076 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-06-03T18:41:34.087 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-03T18:41:34.214 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-03T18:41:34.215 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-03T18:41:34.343 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-06-03T18:41:34.344 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-03T18:41:34.484 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-03T18:41:34.495 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-03T18:41:34.634 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-03T18:41:34.635 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-03T18:41:34.771 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-03T18:41:34.782 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-03T18:41:34.923 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-03T18:41:35.113 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-03T18:41:35.181 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-03T18:41:35.271 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-03T18:41:35.282 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-03T18:41:35.400 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-03T18:41:35.411 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-03T18:41:35.547 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-03T18:41:35.548 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-06-03T18:41:35.682 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-03T18:41:35.684 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-03T18:41:35.812 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-03T18:41:35.813 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-03T18:41:35.948 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-03T18:41:35.959 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-03T18:41:36.087 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-03T18:41:36.098 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-03T18:41:36.233 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-03T18:41:36.242 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-03T18:41:36.383 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-06-03T18:41:36.384 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-03T18:41:36.516 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-03T18:41:36.517 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-03T18:41:36.655 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-03T18:41:36.667 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-03T18:41:36.793 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-03T18:41:36.814 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-03T18:41:36.921 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-03T18:41:36.944 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-03T18:41:37.076 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-03T18:41:37.088 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-03T18:41:37.213 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-03T18:41:37.226 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-03T18:41:37.357 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-03T18:41:37.369 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-03T18:41:37.495 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-03T18:41:37.496 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-03T18:41:37.627 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-03T18:41:37.638 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-03T18:41:37.764 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-03T18:41:37.776 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-03T18:41:37.903 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-03T18:41:37.914 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-03T18:41:38.047 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-03T18:41:38.063 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-03T18:41:38.189 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-03T18:41:38.221 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-03T18:41:38.326 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-03T18:41:38.368 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-03T18:41:38.458 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-03T18:41:38.521 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-03T18:41:38.604 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-03T18:41:38.667 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-03T18:41:38.739 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-03T18:41:38.819 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-03T18:41:38.948 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-03T18:41:39.093 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-03T18:41:39.234 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-03T18:41:39.383 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-03T18:41:39.539 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-03T18:41:39.675 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-03T18:41:39.816 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-03T18:41:39.882 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-03T18:41:39.966 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-03T18:41:40.019 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-03T18:41:40.113 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-03T18:41:40.161 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-03T18:41:40.256 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-03T18:41:40.309 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-03T18:41:40.402 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-03T18:41:40.444 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-03T18:41:40.537 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-03T18:41:40.580 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-03T18:41:40.673 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-03T18:41:40.716 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-03T18:41:40.810 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-03T18:41:40.843 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-03T18:41:40.954 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-03T18:41:40.975 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-03T18:41:41.121 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-03T18:41:41.126 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-03T18:41:41.287 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-03T18:41:41.309 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-03T18:41:41.446 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-03T18:41:41.478 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-03T18:41:41.604 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-06-03T18:41:41.633 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-03T18:41:41.758 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-03T18:41:41.794 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-03T18:41:41.948 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-03T18:41:42.083 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-03T18:41:42.095 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-03T18:41:42.220 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-03T18:41:42.243 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-03T18:41:42.354 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-03T18:41:42.387 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-03T18:41:42.483 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-03T18:41:42.526 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-03T18:41:42.621 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-03T18:41:42.674 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-06-03T18:41:42.759 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-03T18:41:42.812 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-03T18:41:42.896 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-03T18:41:42.949 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-03T18:41:43.048 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-03T18:41:43.092 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-03T18:41:43.206 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-03T18:41:43.235 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-03T18:41:43.357 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-06-03T18:41:43.380 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-03T18:41:43.495 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-03T18:41:43.527 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-06-03T18:41:43.682 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-06-03T18:41:43.817 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-06-03T18:41:43.951 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-03T18:41:44.087 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-03T18:41:44.233 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-03T18:41:44.368 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-03T18:41:44.504 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-03T18:41:44.650 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-03T18:41:44.652 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-06-03T18:41:44.785 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-03T18:41:44.811 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-03T18:41:44.928 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-03T18:41:44.940 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-03T18:41:45.064 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-03T18:41:45.065 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-03T18:41:45.201 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-03T18:41:45.202 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-03T18:41:45.332 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-06-03T18:41:45.355 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-03T18:41:45.480 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-03T18:41:45.495 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-03T18:41:45.621 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-03T18:41:45.651 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-03T18:41:45.751 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-03T18:41:45.774 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-03T18:41:45.887 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-03T18:41:45.909 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-03T18:41:46.037 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-03T18:41:46.049 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-03T18:41:46.175 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-06-03T18:41:46.321 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-06-03T18:41:46.457 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-06-03T18:41:46.613 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-03T18:41:46.675 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-03T18:41:46.748 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-03T18:41:46.884 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-03T18:41:47.019 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-03T18:41:47.154 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-03T18:41:47.196 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "name": "prateek", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-06-03T18:41:47.289 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-03T18:41:47.433 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-03T18:41:47.567 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-03T18:41:47.651 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "name": "jelopez", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-06-03T18:41:47.719 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-03T18:41:47.757 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:47 +0000 (0:00:17.513) 0:00:26.722 ******** 2020-06-03T18:41:47.790 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-06-03T18:41:47.829 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:47 +0000 (0:00:00.070) 0:00:26.792 ******** 2020-06-03T18:41:47.837 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:47.839 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-03T18:41:47.883 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-06-03T18:41:47.887 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-03T18:41:47.889 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-03T18:41:47.890 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-03T18:41:47.892 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-03T18:41:47.893 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-03T18:41:47.895 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-03T18:41:47.899 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-03T18:41:47.901 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-06-03T18:41:47.902 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-06-03T18:41:47.904 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-06-03T18:41:47.905 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-06-03T18:41:47.907 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}]}, "changed": 2020-06-03T18:41:47.909 INFO:teuthology.task.ansible.out: false} 2020-06-03T18:41:47.943 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:47 +0000 (0:00:00.116) 0:00:26.909 ******** 2020-06-03T18:41:47.963 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:47.964 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-03T18:41:48.097 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-03T18:41:48.234 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-03T18:41:48.373 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-03T18:41:48.541 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-03T18:41:48.669 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-03T18:41:48.777 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-06-03T18:41:48.779 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com -> localhost] => {"after": "36acb243999edacffcb85d4e38b209d6e3ce3636", "attempts": 1, "before": "36acb243999edacffcb85d4e38b209d6e3ce3636", "changed": false, "remote_url_changed": false} 2020-06-03T18:41:48.820 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:48 +0000 (0:00:00.876) 0:00:27.785 ******** 2020-06-03T18:41:49.018 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:49.025 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-03T18:41:49.155 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-03T18:41:49.281 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-03T18:41:49.455 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:49.457 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-03T18:41:49.600 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-03T18:41:49.602 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-03T18:41:49.723 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:49.725 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "name": "prateek", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-06-03T18:41:49.745 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:49.747 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-03T18:41:49.910 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-03T18:41:49.912 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-03T18:41:50.044 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-03T18:41:50.188 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-03T18:41:50.263 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-06-03T18:41:50.264 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "name": "jelopez", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-06-03T18:41:50.340 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:50.342 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1s 2020-06-03T18:41:50.345 INFO:teuthology.task.ansible.out:cr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:50.376 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:50 +0000 (0:00:01.556) 0:00:29.341 ******** 2020-06-03T18:41:50.410 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-06-03T18:41:50.446 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:50 +0000 (0:00:00.070) 0:00:29.412 ******** 2020-06-03T18:41:50.477 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:50.479 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-03T18:41:50.481 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:50.490 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:50.502 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-06-03T18:41:50.505 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-03T18:41:50.507 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-03T18:41:50.508 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-03T18:41:50.510 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-03T18:41:50.511 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-03T18:41:50.513 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-03T18:41:50.514 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4Uk 2020-06-03T18:41:50.515 INFO:teuthology.task.ansible.out:ibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-03T18:41:50.517 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "beng 2020-06-03T18:41:50.518 INFO:teuthology.task.ansible.out:land@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name" 2020-06-03T18:41:50.520 INFO:teuthology.task.ansible.out:: "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643 2020-06-03T18:41:50.521 INFO:teuthology.task.ansible.out:eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb 2020-06-03T18:41:50.523 INFO:teuthology.task.ansible.out:801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}]}, "changed": false} TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:50.557 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:50 +0000 (0:00:00.111) 0:00:29.523 ******** 2020-06-03T18:41:50.669 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-03T18:41:50.807 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-03T18:41:50.809 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-03T18:41:50.952 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-03T18:41:51.108 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-03T18:41:51.233 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi112.front.sepia.ceph.com -> localhost] => {"after": "36acb243999edacffcb85d4e38b209d6e3ce3636", "attempts": 1, "before": "36acb243999edacffcb85d4e38b209d6e3ce3636", "changed": false, "remote_url_changed": false} 2020-06-03T18:41:51.275 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:41:51 +0000 (0:00:00.717) 0:00:30.241 ******** 2020-06-03T18:41:51.284 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:41:51.286 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-03T18:41:51.379 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-03T18:41:51.465 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-03T18:41:51.517 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-03T18:41:51.606 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-03T18:41:51.659 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-03T18:41:51.788 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-03T18:41:51.799 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-03T18:41:51.801 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-03T18:41:51.935 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-03T18:41:51.937 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-03T18:41:51.946 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-03T18:41:51.948 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-03T18:41:52.071 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-03T18:41:52.074 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-03T18:41:52.207 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-03T18:41:52.209 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-03T18:41:52.211 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-03T18:41:52.343 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-03T18:41:52.345 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-03T18:41:52.474 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-03T18:41:52.486 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-03T18:41:52.600 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-03T18:41:52.622 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-06-03T18:41:52.634 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:52.737 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-03T18:41:52.759 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-03T18:41:52.761 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-06-03T18:41:52.770 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:52.772 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:52.782 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:41:52.865 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-03T18:41:52.928 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-03T18:41:52.998 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-03T18:41:53.070 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-03T18:41:53.073 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-03T18:41:53.133 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-03T18:41:53.135 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-03T18:41:53.196 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-03T18:41:53.268 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-03T18:41:53.331 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-03T18:41:53.393 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-03T18:41:53.456 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-03T18:41:53.458 INFO:teuthology.task.ansible.out: 2020-06-03T18:41:53.519 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-03T18:41:53.581 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-03T18:41:53.654 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-03T18:41:53.717 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-03T18:41:53.789 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-03T18:41:53.842 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-03T18:41:53.915 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-03T18:41:53.973 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-03T18:41:53.975 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-03T18:41:54.046 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-03T18:41:54.099 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-03T18:41:54.101 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-03T18:41:54.182 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-03T18:41:54.237 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-03T18:41:54.310 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-06-03T18:41:54.362 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-03T18:41:54.451 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-06-03T18:41:54.453 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-03T18:41:54.494 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-03T18:41:54.496 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-06-03T18:41:54.608 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-03T18:41:54.610 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-03T18:41:54.619 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-03T18:41:54.743 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-03T18:41:54.754 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-03T18:41:54.888 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-03T18:41:54.890 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-03T18:41:55.023 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-03T18:41:55.034 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-03T18:41:55.148 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-03T18:41:55.170 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-03T18:41:55.171 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-03T18:41:55.273 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-03T18:41:55.275 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-03T18:41:55.314 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-03T18:41:55.316 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-03T18:41:55.318 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-03T18:41:55.397 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-03T18:41:55.454 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-03T18:41:55.528 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-03T18:41:55.590 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-03T18:41:55.654 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-03T18:41:55.726 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-03T18:41:55.779 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-03T18:41:55.862 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-03T18:41:55.914 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-03T18:41:55.997 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-03T18:41:56.039 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-03T18:41:56.126 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-03T18:41:56.128 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-03T18:41:56.169 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-03T18:41:56.262 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-03T18:41:56.299 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-03T18:41:56.402 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-03T18:41:56.631 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop 2020-06-03T18:41:56.633 INFO:teuthology.task.ansible.out:_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gabrioux", "validate_certs": true} ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e3 2020-06-03T18:41:56.634 INFO:teuthology.task.ansible.out:7ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, " 2020-06-03T18:41:56.636 INFO:teuthology.task.ansible.out:path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-03T18:41:56.677 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-03T18:41:56.741 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-03T18:41:56.743 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-03T18:41:56.824 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-03T18:41:56.880 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-03T18:41:56.953 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-03T18:41:56.955 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-06-03T18:41:57.025 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-03T18:41:57.088 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-03T18:41:57.161 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-03T18:41:57.224 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-03T18:41:57.327 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-03T18:41:57.329 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-03T18:41:57.359 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-03T18:41:57.453 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-03T18:41:57.455 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-03T18:41:57.457 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-03T18:41:57.495 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-03T18:41:57.597 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-03T18:41:57.629 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-03T18:41:57.734 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-03T18:41:57.777 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-03T18:41:57.871 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-03T18:41:57.923 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-03T18:41:58.006 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-03T18:41:58.059 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-03T18:41:58.132 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-03T18:41:58.195 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-03T18:41:58.259 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-03T18:41:58.260 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-03T18:41:58.322 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-03T18:41:58.382 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-03T18:41:58.455 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-03T18:41:58.530 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-03T18:41:58.613 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-03T18:41:58.671 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-06-03T18:41:58.673 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-06-03T18:41:58.746 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-03T18:41:58.802 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-03T18:41:58.885 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-06-03T18:41:58.941 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-03T18:41:59.035 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-03T18:41:59.067 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-03T18:41:59.069 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-06-03T18:41:59.163 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-03T18:41:59.205 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-03T18:41:59.311 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-03T18:41:59.332 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-03T18:41:59.447 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-03T18:41:59.458 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-03T18:41:59.583 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-03T18:41:59.596 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-03T18:41:59.722 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-03T18:41:59.725 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-03T18:41:59.860 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-03T18:41:59.862 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-03T18:41:59.981 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-03T18:42:00.013 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-03T18:42:00.109 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-03T18:42:00.151 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-03T18:42:00.263 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-03T18:42:00.287 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-03T18:42:00.376 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-03T18:42:00.444 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-03T18:42:00.508 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-03T18:42:00.584 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-03T18:42:00.586 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-03T18:42:00.638 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-03T18:42:00.723 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-03T18:42:00.725 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-03T18:42:00.766 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-03T18:42:00.853 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-03T18:42:00.923 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-06-03T18:42:00.987 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-03T18:42:01.066 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-03T18:42:01.129 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-03T18:42:01.202 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-03T18:42:01.265 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-03T18:42:01.327 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-03T18:42:01.400 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-03T18:42:01.462 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-03T18:42:01.535 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-03T18:42:01.587 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-03T18:42:01.672 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-06-03T18:42:01.725 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-03T18:42:01.798 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-03T18:42:01.851 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-03T18:42:01.944 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-03T18:42:01.975 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-03T18:42:02.069 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-03T18:42:02.100 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-03T18:42:02.203 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-03T18:42:02.237 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-03T18:42:02.340 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-03T18:42:02.362 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-03T18:42:02.466 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-03T18:42:02.488 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-03T18:42:02.490 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-03T18:42:02.602 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-03T18:42:02.628 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-03T18:42:02.630 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-03T18:42:02.732 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-06-03T18:42:02.734 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-03T18:42:02.753 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-03T18:42:02.858 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-03T18:42:02.880 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-03T18:42:02.994 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-03T18:42:03.026 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-03T18:42:03.128 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-06-03T18:42:03.150 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-03T18:42:03.262 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-03T18:42:03.304 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-03T18:42:03.387 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-03T18:42:03.449 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-03T18:42:03.522 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-03T18:42:03.584 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-06-03T18:42:03.659 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-03T18:42:03.711 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-03T18:42:03.783 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-06-03T18:42:03.853 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-03T18:42:03.927 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-06-03T18:42:03.980 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-03T18:42:04.050 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-06-03T18:42:04.112 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-03T18:42:04.185 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-06-03T18:42:04.238 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-03T18:42:04.312 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-06-03T18:42:04.376 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-03T18:42:04.439 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-03T18:42:04.517 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-03T18:42:04.560 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-03T18:42:04.653 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-06-03T18:42:04.655 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-03T18:42:04.697 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-03T18:42:04.782 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-03T18:42:04.828 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-03T18:42:04.916 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:04 +0000 (0:00:13.640) 0:00:43.882 ******** 2020-06-03T18:42:04.934 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-03T18:42:04.935 INFO:teuthology.task.ansible.out: 2020-06-03T18:42:04.979 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:04.980 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:04.983 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:04.984 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:04.994 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:04.996 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.007 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.008 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.020 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.031 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.063 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.064 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-06-03T18:42:05.160 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:05.161 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-06-03T18:42:05.172 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.204 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.205 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-03T18:42:05.300 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:05.301 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-03T18:42:05.333 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.334 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-03T18:42:05.438 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:05.439 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-03T18:42:05.461 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.462 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-03T18:42:05.558 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:05.559 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-06-03T18:42:05.569 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.570 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.581 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.582 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.593 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.594 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.594 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-03T18:42:05.605 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:05.606 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.616 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.627 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.628 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.642 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.653 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.655 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.666 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.667 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.677 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.679 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.688 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.699 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.701 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.711 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.722 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.724 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.733 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:05.735 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.735 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfBy 2020-06-03T18:42:05.737 INFO:teuthology.task.ansible.out:eEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.746 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.747 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.757 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.769 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.771 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.780 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.792 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.794 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.803 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.804 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.815 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.826 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.827 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.837 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.848 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.850 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.859 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.871 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.872 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.883 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.885 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.896 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.907 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.909 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.919 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.930 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.931 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.941 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.942 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.963 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.965 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.974 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.985 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.986 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.997 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:05.998 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.008 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:06.010 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.021 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.022 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.032 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.043 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.045 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.054 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.056 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.066 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.076 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.078 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.088 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.088 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.099 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.110 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.112 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.121 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.132 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.134 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.145 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.156 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.158 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.167 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.169 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.189 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.191 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.201 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.212 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.213 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.223 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.224 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.235 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.246 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.248 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.257 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.268 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.270 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.281 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.282 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.292 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.303 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.305 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.363 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:06 +0000 (0:00:01.446) 0:00:45.328 ******** 2020-06-03T18:42:06.404 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-06-03T18:42:06.406 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-03T18:42:06.446 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:06 +0000 (0:00:00.083) 0:00:45.411 ******** 2020-06-03T18:42:06.535 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-03T18:42:06.628 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:06.630 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-03T18:42:06.671 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:06.673 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-03T18:42:06.767 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:06.769 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-03T18:42:06.811 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-03T18:42:06.813 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-03T18:42:06.914 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:06 +0000 (0:00:00.468) 0:00:45.880 ******** 2020-06-03T18:42:06.933 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:06.935 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-06-03T18:42:06.980 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:06.982 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.993 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:06.994 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.004 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.015 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.017 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.028 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-03T18:42:07.030 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.039 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.154 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-03T18:42:07.186 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 2020-06-03T18:42:07.188 INFO:teuthology.task.ansible.out: 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.279 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-03T18:42:07.321 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-03T18:42:07.414 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:07.415 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-03T18:42:07.448 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:07.449 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-03T18:42:07.547 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:07.548 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-06-03T18:42:07.583 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:07.584 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-06-03T18:42:07.595 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.607 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.618 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.640 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.641 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.653 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.664 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.667 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.678 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.680 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:07.681 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.683 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.693 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.705 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.706 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.717 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.719 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.730 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.731 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.743 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.754 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.755 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.767 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.768 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.779 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.780 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.792 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.794 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.805 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.806 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:07.808 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.819 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.831 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.832 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.843 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.845 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.856 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.868 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.869 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.881 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.893 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.894 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.906 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.907 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.918 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.930 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.931 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.943 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:07.944 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-03T18:42:07.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.948 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.958 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.970 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.972 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.984 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.985 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:07.997 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.008 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.010 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.021 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.023 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.035 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.036 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.047 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.059 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.061 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.071 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:08.072 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.082 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.083 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.094 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.105 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.116 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.118 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.127 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.139 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.140 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.153 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.164 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.165 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.175 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.186 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.196 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:08.198 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-03T18:42:08.208 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-03T18:42:08.209 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.230 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.241 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.243 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.253 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.255 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.265 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.266 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.276 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.286 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.288 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.298 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.308 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.310 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.320 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.322 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi061.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.332 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.343 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.345 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.357 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.368 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.378 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.380 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.390 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.439 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:08 +0000 (0:00:01.524) 0:00:47.404 ******** 2020-06-03T18:42:08.459 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:08.461 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-03T18:42:08.471 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:08.506 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:08 +0000 (0:00:00.067) 0:00:47.471 ******** 2020-06-03T18:42:08.578 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:08.580 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-03T18:42:08.682 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-03T18:42:08.704 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-03T18:42:08.824 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-03T18:42:08.835 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-03T18:42:08.959 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-06-03T18:42:08.961 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-03T18:42:09.095 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-03T18:42:09.097 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-03T18:42:09.225 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-03T18:42:09.226 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-03T18:42:09.350 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-03T18:42:09.361 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-03T18:42:09.475 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-03T18:42:09.486 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-03T18:42:09.615 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-03T18:42:09.617 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-06-03T18:42:09.739 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-06-03T18:42:09.751 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-03T18:42:09.874 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-06-03T18:42:09.875 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-03T18:42:10.001 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-03T18:42:10.003 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-03T18:42:10.136 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-06-03T18:42:10.138 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-03T18:42:10.261 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-03T18:42:10.262 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-03T18:42:10.408 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-06-03T18:42:10.410 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-03T18:42:10.554 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-03T18:42:10.555 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-03T18:42:10.680 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-03T18:42:10.682 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-03T18:42:10.808 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-03T18:42:10.819 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-03T18:42:10.956 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-03T18:42:10.968 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-03T18:42:11.083 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-03T18:42:11.085 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-03T18:42:11.212 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-03T18:42:11.223 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-03T18:42:11.337 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-03T18:42:11.348 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-03T18:42:11.472 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-03T18:42:11.472 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-03T18:42:11.607 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-03T18:42:11.608 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-03T18:42:11.734 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-03T18:42:11.736 INFO:teuthology.task.ansible.out: ok: [smithi061.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-03T18:42:11.865 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-03T18:42:11.866 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-03T18:42:12.001 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-03T18:42:12.004 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-03T18:42:12.137 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-03T18:42:12.155 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-03T18:42:12.196 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:03.689) 0:00:51.161 ******** 2020-06-03T18:42:12.267 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-03T18:42:12.391 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-03T18:42:12.527 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-03T18:42:12.579 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-03T18:42:12.581 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "checksum": "3f67a1aec94fcdb4f49608a4758f290e7aa48f9d", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ca2bb5759342081aa54fc8ea54899674", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591209732.2224646-219162661905983/source", "state": "file", "uid": 0} 2020-06-03T18:42:12.618 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.422) 0:00:51.584 ******** 2020-06-03T18:42:12.648 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-06-03T18:42:12.650 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => { "msg": "Host smithi061.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-06-03T18:42:12.690 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.072) 0:00:51.656 ******** 2020-06-03T18:42:12.700 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:12.702 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-03T18:42:12.733 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-06-03T18:42:12.735 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-06-03T18:42:12.772 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.081) 0:00:51.738 ******** 2020-06-03T18:42:12.793 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:12.795 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-03T18:42:12.804 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-06-03T18:42:12.806 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:12.842 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.068) 0:00:51.806 ******** 2020-06-03T18:42:12.871 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-06-03T18:42:12.873 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:12.908 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.067) 0:00:51.874 ******** 2020-06-03T18:42:12.928 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:12.930 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-03T18:42:12.940 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-06-03T18:42:12.942 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:12.977 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:12 +0000 (0:00:00.068) 0:00:51.943 ******** 2020-06-03T18:42:13.007 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-06-03T18:42:13.010 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.045 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.067) 0:00:52.010 ******** 2020-06-03T18:42:13.057 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi112.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-03T18:42:13.077 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.115 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.070) 0:00:52.080 ******** 2020-06-03T18:42:13.145 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-06-03T18:42:13.146 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.185 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.070) 0:00:52.151 ******** 2020-06-03T18:42:13.194 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.196 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-03T18:42:13.217 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-06-03T18:42:13.219 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.255 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.069) 0:00:52.220 ******** 2020-06-03T18:42:13.285 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-06-03T18:42:13.287 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.323 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.068) 0:00:52.289 ******** 2020-06-03T18:42:13.332 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.334 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-03T18:42:13.365 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-06-03T18:42:13.367 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:13.402 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.078) 0:00:52.368 ******** 2020-06-03T18:42:13.434 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.436 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-03T18:42:13.445 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-06-03T18:42:13.447 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.483 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.080) 0:00:52.449 ******** 2020-06-03T18:42:13.524 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-06-03T18:42:13.526 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.564 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.080) 0:00:52.530 ******** 2020-06-03T18:42:13.584 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.586 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-03T18:42:13.596 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-03T18:42:13.598 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:13.636 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.072) 0:00:52.602 ******** 2020-06-03T18:42:13.675 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-06-03T18:42:13.677 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.713 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.076) 0:00:52.678 ******** 2020-06-03T18:42:13.723 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.724 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-03T18:42:13.745 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:13.781 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.067) 0:00:52.746 ******** 2020-06-03T18:42:13.820 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-06-03T18:42:13.822 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.857 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.076) 0:00:52.823 ******** 2020-06-03T18:42:13.865 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:13.867 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-03T18:42:13.898 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.934 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.076) 0:00:52.899 ******** 2020-06-03T18:42:13.963 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-06-03T18:42:13.965 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:13.999 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:13 +0000 (0:00:00.065) 0:00:52.964 ******** 2020-06-03T18:42:14.010 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi112.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-03T18:42:14.032 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.068 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.068) 0:00:53.033 ******** 2020-06-03T18:42:14.097 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:14.099 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-03T18:42:14.109 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:14.149 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.081) 0:00:53.115 ******** 2020-06-03T18:42:14.179 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-03T18:42:14.181 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.215 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.066) 0:00:53.181 ******** 2020-06-03T18:42:14.235 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-03T18:42:14.237 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-03T18:42:14.252 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-03T18:42:14.254 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.292 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.076) 0:00:53.257 ******** 2020-06-03T18:42:14.329 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.037) 0:00:53.294 ******** 2020-06-03T18:42:14.336 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-06-03T18:42:14.338 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.374 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.045) 0:00:53.340 ******** 2020-06-03T18:42:14.402 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.441 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.066) 0:00:53.406 ******** 2020-06-03T18:42:14.471 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-06-03T18:42:14.473 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.507 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.066) 0:00:53.472 ******** 2020-06-03T18:42:14.538 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-06-03T18:42:14.539 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.575 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.067) 0:00:53.540 ******** 2020-06-03T18:42:14.605 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-06-03T18:42:14.607 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:14.642 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.067) 0:00:53.607 ******** 2020-06-03T18:42:14.673 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-06-03T18:42:14.674 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.713 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.071) 0:00:53.678 ******** 2020-06-03T18:42:14.723 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-03T18:42:14.725 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "checksum": "3f67a1aec94fcdb4f49608a4758f290e7aa48f9d", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ca2bb5759342081aa54fc8ea54899674", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591209734.3580046-124806071050949/source", "state": "file", "uid": 0} 2020-06-03T18:42:14.761 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.047) 0:00:53.726 ******** 2020-06-03T18:42:14.770 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-06-03T18:42:14.772 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.808 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.047) 0:00:53.774 ******** 2020-06-03T18:42:14.819 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-06-03T18:42:14.821 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => { "msg": "Host smithi112.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-06-03T18:42:14.857 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.048) 0:00:53.822 ******** 2020-06-03T18:42:14.866 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-03T18:42:14.868 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:14.909 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.052) 0:00:53.875 ******** 2020-06-03T18:42:14.917 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-06-03T18:42:14.919 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-06-03T18:42:14.955 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.045) 0:00:53.920 ******** 2020-06-03T18:42:14.962 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-06-03T18:42:14.964 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.002 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:14 +0000 (0:00:00.046) 0:00:53.967 ******** 2020-06-03T18:42:15.010 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-06-03T18:42:15.012 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.048 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.013 ******** 2020-06-03T18:42:15.054 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-06-03T18:42:15.057 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.092 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.044) 0:00:54.058 ******** 2020-06-03T18:42:15.102 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-06-03T18:42:15.104 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.142 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.049) 0:00:54.107 ******** 2020-06-03T18:42:15.150 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-03T18:42:15.151 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.190 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.048) 0:00:54.155 ******** 2020-06-03T18:42:15.198 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-06-03T18:42:15.200 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.235 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.201 ******** 2020-06-03T18:42:15.263 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-06-03T18:42:15.265 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.299 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.063) 0:00:54.264 ******** 2020-06-03T18:42:15.333 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-06-03T18:42:15.335 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.368 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.069) 0:00:54.334 ******** 2020-06-03T18:42:15.380 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-03T18:42:15.382 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.010073", "end": "2020-06-03 18:42:15.035357", "rc": 0, "start": "2020-06-03 18:42:15.025284", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-03T18:42:15.417 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.049) 0:00:54.383 ******** 2020-06-03T18:42:15.427 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-06-03T18:42:15.429 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.463 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.046) 0:00:54.429 ******** 2020-06-03T18:42:15.474 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-06-03T18:42:15.475 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.511 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.048) 0:00:54.477 ******** 2020-06-03T18:42:15.524 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.558 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.046) 0:00:54.524 ******** 2020-06-03T18:42:15.567 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-06-03T18:42:15.569 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.605 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.046) 0:00:54.571 ******** 2020-06-03T18:42:15.614 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-06-03T18:42:15.616 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.650 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.616 ******** 2020-06-03T18:42:15.658 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-03T18:42:15.660 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.696 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.662 ******** 2020-06-03T18:42:15.704 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-06-03T18:42:15.706 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:15.746 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.049) 0:00:54.711 ******** 2020-06-03T18:42:15.754 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-06-03T18:42:15.755 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.791 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.757 ******** 2020-06-03T18:42:15.801 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-06-03T18:42:15.803 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.838 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.045) 0:00:54.803 ******** 2020-06-03T18:42:15.846 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-03T18:42:15.848 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.884 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.047) 0:00:54.850 ******** 2020-06-03T18:42:15.895 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.936 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.051) 0:00:54.902 ******** 2020-06-03T18:42:15.947 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-06-03T18:42:15.949 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:15.986 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:15 +0000 (0:00:00.049) 0:00:54.951 ******** 2020-06-03T18:42:15.996 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-03T18:42:15.998 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:16.035 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.049) 0:00:55.001 ******** 2020-06-03T18:42:16.043 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-06-03T18:42:16.045 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.085 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.049) 0:00:55.050 ******** 2020-06-03T18:42:16.204 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.046) 0:00:55.097 ******** TASK [common : Register with subscription-manager.] **************************** skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:16.209 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.077) 0:00:55.175 ******** 2020-06-03T18:42:16.249 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-06-03T18:42:16.250 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.293 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.083) 0:00:55.258 ******** 2020-06-03T18:42:16.332 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-06-03T18:42:16.334 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.385 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.091) 0:00:55.350 ******** 2020-06-03T18:42:16.426 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-06-03T18:42:16.429 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.473 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.088) 0:00:55.438 ******** 2020-06-03T18:42:16.501 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-06-03T18:42:16.503 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.537 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.064) 0:00:55.502 ******** 2020-06-03T18:42:16.582 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-06-03T18:42:16.584 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:16.617 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.080) 0:00:55.582 ******** 2020-06-03T18:42:16.647 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-03T18:42:16.649 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.684 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.066) 0:00:55.649 ******** 2020-06-03T18:42:16.713 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-03T18:42:16.715 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.753 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.069) 0:00:55.719 ******** 2020-06-03T18:42:16.798 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-06-03T18:42:16.800 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.835 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.081) 0:00:55.800 ******** 2020-06-03T18:42:16.844 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-06-03T18:42:16.845 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.581159", "end": "2020-06-03 18:42:16.500424", "rc": 0, "start": "2020-06-03 18:42:15.919265", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:16.885 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.050) 0:00:55.851 ******** 2020-06-03T18:42:16.895 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-06-03T18:42:16.897 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.934 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.048) 0:00:55.899 ******** 2020-06-03T18:42:16.944 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-06-03T18:42:16.946 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:16.982 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:16 +0000 (0:00:00.048) 0:00:55.947 ******** 2020-06-03T18:42:16.991 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-06-03T18:42:16.993 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.035 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.053) 0:00:56.000 ******** 2020-06-03T18:42:17.064 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-06-03T18:42:17.066 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.099 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.063) 0:00:56.064 ******** 2020-06-03T18:42:17.127 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-06-03T18:42:17.129 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:17.169 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.069) 0:00:56.134 ******** 2020-06-03T18:42:17.177 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-06-03T18:42:17.178 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.025022", "end": "2020-06-03 18:42:16.839126", "rc": 0, "start": "2020-06-03 18:42:16.814104", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:17.215 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.046) 0:00:56.181 ******** 2020-06-03T18:42:17.225 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-06-03T18:42:17.227 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.265 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.049) 0:00:56.231 ******** 2020-06-03T18:42:17.294 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-06-03T18:42:17.296 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.331 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.066) 0:00:56.297 ******** 2020-06-03T18:42:17.370 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-03T18:42:17.372 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.406 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.074) 0:00:56.372 ******** 2020-06-03T18:42:17.456 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-06-03T18:42:17.458 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.494 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.088) 0:00:56.460 ******** 2020-06-03T18:42:17.546 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-06-03T18:42:17.547 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.588 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.093) 0:00:56.553 ******** 2020-06-03T18:42:17.627 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-03T18:42:17.629 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.663 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.075) 0:00:56.629 ******** 2020-06-03T18:42:17.832 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016904", "end": "2020-06-03 18:42:17.423327", "rc": 0, "start": "2020-06-03 18:42:17.406423", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-03T18:42:17.867 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.203) 0:00:56.832 ******** 2020-06-03T18:42:17.905 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-06-03T18:42:17.907 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:17.940 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:17 +0000 (0:00:00.073) 0:00:56.906 ******** 2020-06-03T18:42:17.985 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.018 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.077) 0:00:56.983 ******** 2020-06-03T18:42:18.056 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-03T18:42:18.058 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.091 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.073) 0:00:57.056 ******** 2020-06-03T18:42:18.129 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-06-03T18:42:18.130 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.164 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.073) 0:00:57.130 ******** 2020-06-03T18:42:18.193 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-03T18:42:18.195 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.228 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.063) 0:00:57.193 ******** 2020-06-03T18:42:18.256 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-06-03T18:42:18.258 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.291 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.063) 0:00:57.257 ******** 2020-06-03T18:42:18.326 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:18.354 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.062) 0:00:57.320 ******** 2020-06-03T18:42:18.902 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-06-03T18:42:18.904 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.413126", "end": "2020-06-03 18:42:18.500442", "rc": 0, "start": "2020-06-03 18:42:18.087316", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:18.938 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.583) 0:00:57.903 ******** 2020-06-03T18:42:18.967 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-06-03T18:42:18.968 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:19.002 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:18 +0000 (0:00:00.063) 0:00:57.967 ******** 2020-06-03T18:42:19.164 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-06-03T18:42:19.165 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.020701", "end": "2020-06-03 18:42:18.760327", "rc": 0, "start": "2020-06-03 18:42:18.739626", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:19.200 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:19 +0000 (0:00:00.197) 0:00:58.165 ******** 2020-06-03T18:42:20.328 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-06-03T18:42:20.330 INFO:teuthology.task.ansible.out: [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:42:20.368 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:20 +0000 (0:00:01.167) 0:00:59.333 ******** 2020-06-03T18:42:21.274 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-06-03T18:42:21.276 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:42:21.320 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:21 +0000 (0:00:00.951) 0:01:00.285 ******** 2020-06-03T18:42:21.348 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-06-03T18:42:21.350 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:21.386 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:21 +0000 (0:00:00.066) 0:01:00.352 ******** 2020-06-03T18:42:22.048 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-06-03T18:42:22.050 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-06-03T18:42:22.095 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:22 +0000 (0:00:00.708) 0:01:01.060 ******** 2020-06-03T18:42:22.105 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-06-03T18:42:22.107 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:42:22.158 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:22 +0000 (0:00:00.062) 0:01:01.123 ******** 2020-06-03T18:42:23.004 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-06-03T18:42:23.005 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:42:23.040 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:23 +0000 (0:00:00.882) 0:01:02.006 ******** 2020-06-03T18:42:23.055 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-06-03T18:42:23.057 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:42:23.097 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:23 +0000 (0:00:00.056) 0:01:02.062 ******** 2020-06-03T18:42:23.136 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-06-03T18:42:23.138 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:23.173 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:23 +0000 (0:00:00.073) 0:01:02.136 ******** 2020-06-03T18:42:23.540 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-06-03T18:42:23.542 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-06-03T18:42:23.581 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:23 +0000 (0:00:00.410) 0:01:02.547 ******** 2020-06-03T18:42:23.609 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-03T18:42:23.611 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-06-03T18:42:23.960 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-06-03T18:42:24.267 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-06-03T18:42:24.333 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:24 +0000 (0:00:00.751) 0:01:03.298 ******** 2020-06-03T18:42:24.404 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-06-03T18:42:24.406 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:42:24.444 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:24 +0000 (0:00:00.111) 0:01:03.409 ******** 2020-06-03T18:42:24.473 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-06-03T18:42:24.475 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:42:24.510 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:24 +0000 (0:00:00.066) 0:01:03.476 ******** 2020-06-03T18:42:24.804 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi112.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-06-03T18:42:24.839 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-03T18:42:24.841 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-06-03T18:42:25.159 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-03T18:42:25.160 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-06-03T18:42:25.191 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-03T18:42:25.193 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-06-03T18:42:25.463 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-03T18:42:25.465 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi061.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-06-03T18:42:25.466 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-03T18:42:25.467 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-03T18:42:25.512 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:01.001) 0:01:04.477 ******** 2020-06-03T18:42:25.556 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.044) 0:01:04.522 ******** 2020-06-03T18:42:25.565 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-03T18:42:25.567 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-06-03T18:42:25.604 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.047) 0:01:04.570 ******** 2020-06-03T18:42:25.636 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-06-03T18:42:25.638 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.672 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.067) 0:01:04.638 ******** 2020-06-03T18:42:25.690 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-06-03T18:42:25.692 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:42:25.726 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.053) 0:01:04.691 ******** 2020-06-03T18:42:25.734 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-06-03T18:42:25.736 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:25.776 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.050) 0:01:04.741 ******** 2020-06-03T18:42:25.815 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-06-03T18:42:25.817 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.826 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.837 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.848 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.884 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.108) 0:01:04.849 ******** 2020-06-03T18:42:25.917 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:25.949 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:25 +0000 (0:00:00.064) 0:01:04.914 ******** 2020-06-03T18:42:26.081 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-03T18:42:26.083 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-06-03T18:42:26.384 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-06-03T18:42:26.684 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-06-03T18:42:26.733 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:26 +0000 (0:00:00.784) 0:01:05.699 ******** 2020-06-03T18:42:26.774 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-03T18:42:26.776 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-06-03T18:42:26.811 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:26 +0000 (0:00:00.078) 0:01:05.777 ******** 2020-06-03T18:42:26.841 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-06-03T18:42:26.843 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:26.896 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:26 +0000 (0:00:00.085) 0:01:05.862 ******** 2020-06-03T18:42:26.926 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-06-03T18:42:26.928 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:26.962 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:26 +0000 (0:00:00.065) 0:01:05.928 ******** 2020-06-03T18:42:27.012 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-06-03T18:42:27.014 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:27.024 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:27.026 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:27.062 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:27 +0000 (0:00:00.099) 0:01:06.027 ******** 2020-06-03T18:42:27.103 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-06-03T18:42:27.105 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:27.141 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:27 +0000 (0:00:00.078) 0:01:06.106 ******** 2020-06-03T18:42:29.140 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-06-03T18:42:29.142 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n nagios-plugins-basic\n1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.\nNeed to get 6444 B of archives.\nAfter this operation, 0 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]\nFetched 6444 B in 0s (38.9 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137516 files and directories currently installed.)\r\nPreparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...\r\nUnpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...\r\nSetting up nagios-plugins-basic (2.2-3ubuntu3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", " nagios-plugins-basic", "1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.", "Need to get 6444 B of archives.", "After this operation, 0 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]", "Fetched 6444 B in 0s (38.9 kB/s)", "(Reading database ... 2020-06-03T18:42:29.144 INFO:teuthology.task.ansible.out:", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137516 files and directories currently installed.)", "Preparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...", "Unpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...", "Setting up nagios-plugins-basic (2.2-3ubuntu3) ..."]} 2020-06-03T18:42:29.178 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:29 +0000 (0:00:02.037) 0:01:08.144 ******** 2020-06-03T18:42:29.562 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-03T18:42:29.564 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-06-03T18:42:29.610 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:29 +0000 (0:00:00.431) 0:01:08.576 ******** 2020-06-03T18:42:29.947 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-06-03T18:42:29.949 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-06-03T18:42:29.986 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:29 +0000 (0:00:00.375) 0:01:08.951 ******** 2020-06-03T18:42:30.163 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-03T18:42:30.166 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-06-03T18:42:30.254 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-06-03T18:42:30.260 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n nagios-plugins-basic\n1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.\nNeed to get 6444 B of archives.\nAfter this operation, 0 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]\nFetched 6444 B in 0s (39.0 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137516 files and directories currently installed.)\r\nPreparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...\r\nUnpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...\r\nSetting up nagios-plugins-basic (2.2-3ubuntu3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", " nagios-plugins-basic", "1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.", "Need to get 6444 B of archives.", "After this operation, 0 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]", "Fetched 6444 B in 0s (39.0 kB/s)", "(Reading database ... 2020-06-03T18:42:30.262 INFO:teuthology.task.ansible.out:", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137516 files and directories currently installed.)", "Preparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...", "Unpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...", "Setting up nagios-plugins-basic (2.2-3ubuntu3) ..."]} 2020-06-03T18:42:30.306 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.319) 0:01:09.271 ******** 2020-06-03T18:42:30.317 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-03T18:42:30.319 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-06-03T18:42:30.366 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.060) 0:01:09.331 ******** 2020-06-03T18:42:30.396 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-06-03T18:42:30.397 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:30.433 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.067) 0:01:09.399 ******** 2020-06-03T18:42:30.472 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-06-03T18:42:30.474 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:30.511 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.077) 0:01:09.476 ******** 2020-06-03T18:42:30.551 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-06-03T18:42:30.553 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:30.593 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.081) 0:01:09.558 ******** 2020-06-03T18:42:30.665 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-03T18:42:30.667 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-06-03T18:42:30.711 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.118) 0:01:09.677 ******** 2020-06-03T18:42:30.943 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-06-03T18:42:30.945 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-06-03T18:42:30.983 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:30 +0000 (0:00:00.272) 0:01:09.949 ******** 2020-06-03T18:42:31.044 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-06-03T18:42:31.046 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-06-03T18:42:31.098 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.114) 0:01:10.063 ******** 2020-06-03T18:42:31.257 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-03T18:42:31.259 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-06-03T18:42:31.374 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-06-03T18:42:31.411 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.313) 0:01:10.376 ******** 2020-06-03T18:42:31.452 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-06-03T18:42:31.453 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:31.513 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.092) 0:01:10.469 ******** 2020-06-03T18:42:31.545 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi112.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-03T18:42:31.582 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.078) 0:01:10.548 ******** 2020-06-03T18:42:31.624 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-06-03T18:42:31.626 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:31.661 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.078) 0:01:10.626 ******** 2020-06-03T18:42:31.728 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-06-03T18:42:31.729 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ActiveEnterTimestampMonotonic": "18398810", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target system.slice systemd-tmpfiles-setup.service systemd-journald.socket remote-fs.target nss-lookup.target var-run.mount local-fs.target time-sync.target network.target basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:41:04 UTC", "AssertTimestampMonotonic": "18397890", "Before": "xdm.service multi-user.target getty@tty1.service plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ConditionTimestampMonotonic": "18397888", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-06-03T18:42:31.731 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1291", "ExecMainStartTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ExecMainStartTimestampMonotonic": "18398766", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2020-06-03 18:41:04 UTC] ; stop_time=[n/a] ; pid=1291 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:41:04 UTC", "InactiveExitTimestampMonotonic": "18398810", "InvocationID": "14d6ba87e2544430b98494883a41bad2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-06-03T18:42:31.733 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1291", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-06-03T18:42:31.734 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:41:04 UTC", "StateChangeTimestampMonotonic": "18398810", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:41:04 UTC", "WatchdogTimestampMonotonic": "18398806", "WatchdogUSec": "0"}} 2020-06-03T18:42:31.769 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.108) 0:01:10.735 ******** 2020-06-03T18:42:31.800 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-06-03T18:42:31.802 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:31.836 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.066) 0:01:10.802 ******** 2020-06-03T18:42:31.867 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-06-03T18:42:31.868 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:31.913 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.076) 0:01:10.878 ******** 2020-06-03T18:42:31.942 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-06-03T18:42:31.944 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:31.978 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:31 +0000 (0:00:00.065) 0:01:10.944 ******** 2020-06-03T18:42:32.018 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-06-03T18:42:32.020 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-06-03T18:42:32.056 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.077) 0:01:11.022 ******** 2020-06-03T18:42:32.097 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.040) 0:01:11.062 ******** 2020-06-03T18:42:32.105 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-06-03T18:42:32.107 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.144 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.047) 0:01:11.110 ******** 2020-06-03T18:42:32.183 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-06-03T18:42:32.185 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.220 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.075) 0:01:11.185 ******** 2020-06-03T18:42:32.250 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-06-03T18:42:32.252 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.296 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.076) 0:01:11.261 ******** 2020-06-03T18:42:32.337 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-06-03T18:42:32.338 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-06-03T18:42:32.340 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ActiveEnterTimestampMonotonic": "18610050", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target nss-lookup.target basic.target systemd-tmpfiles-setup.service var-run.mount systemd-journald.socket local-fs.target sysinit.target network.target -.mount remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:40:53 UTC", "AssertTimestampMonotonic": "18609081", "Before": "plymouth-quit.service multi-user.target xdm.service getty@tty1.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ConditionTimestampMonotonic": "18609081", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-06-03T18:42:32.342 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1167", "ExecMainStartTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ExecMainStartTimestampMonotonic": "18609969", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2020-06-03 18:40:53 UTC] ; stop_time=[n/a] ; pid=1167 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:40:53 UTC", "InactiveExitTimestampMonotonic": "18610050", "InvocationID": "45813bf701d14483ba6fd8e175be5832", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-06-03T18:42:32.343 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1167", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-06-03T18:42:32.345 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:40:53 UTC", "StateChangeTimestampMonotonic": "18610050", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:40:53 UTC", "WatchdogTimestampMonotonic": "18610049", "WatchdogUSec": "0"}} 2020-06-03T18:42:32.375 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.079) 0:01:11.341 ******** 2020-06-03T18:42:32.427 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.051) 0:01:11.393 ******** 2020-06-03T18:42:32.437 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-06-03T18:42:32.439 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.484 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.056) 0:01:11.449 ******** 2020-06-03T18:42:32.502 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.539 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.054) 0:01:11.504 ******** 2020-06-03T18:42:32.546 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-06-03T18:42:32.548 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.585 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.045) 0:01:11.550 ******** 2020-06-03T18:42:32.596 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-06-03T18:42:32.598 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.634 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.048) 0:01:11.599 ******** 2020-06-03T18:42:32.644 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-06-03T18:42:32.646 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-06-03T18:42:32.683 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.049) 0:01:11.648 ******** 2020-06-03T18:42:32.692 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-06-03T18:42:32.694 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.729 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.046) 0:01:11.695 ******** 2020-06-03T18:42:32.737 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-06-03T18:42:32.739 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-06-03T18:42:32.777 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.047) 0:01:11.743 ******** 2020-06-03T18:42:32.785 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-06-03T18:42:32.787 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.823 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.045) 0:01:11.788 ******** 2020-06-03T18:42:32.830 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-06-03T18:42:32.831 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-06-03T18:42:32.869 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.046) 0:01:11.834 ******** 2020-06-03T18:42:32.877 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-06-03T18:42:32.878 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:32.924 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.052) 0:01:11.886 ******** 2020-06-03T18:42:32.928 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-06-03T18:42:32.930 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-03T18:42:32.976 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:32 +0000 (0:00:00.055) 0:01:11.941 ******** 2020-06-03T18:42:32.984 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-06-03T18:42:32.986 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.021 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.045) 0:01:11.987 ******** 2020-06-03T18:42:33.049 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-06-03T18:42:33.051 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.085 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.063) 0:01:12.051 ******** 2020-06-03T18:42:33.116 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-06-03T18:42:33.118 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-03T18:42:33.152 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.067) 0:01:12.118 ******** 2020-06-03T18:42:33.160 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-06-03T18:42:33.162 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.198 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.046) 0:01:12.164 ******** 2020-06-03T18:42:33.237 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-06-03T18:42:33.239 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.284 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.084) 0:01:12.249 ******** 2020-06-03T18:42:33.303 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-06-03T18:42:33.304 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2020-06-03T18:42:33.357 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.068) 0:01:12.318 ******** 2020-06-03T18:42:33.362 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-06-03T18:42:33.364 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.407 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.052) 0:01:12.370 ******** 2020-06-03T18:42:33.414 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-06-03T18:42:33.416 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.458 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.053) 0:01:12.424 ******** 2020-06-03T18:42:33.468 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-06-03T18:42:33.470 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-06-03T18:42:33.511 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.052) 0:01:12.476 ******** 2020-06-03T18:42:33.528 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:33.568 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.056) 0:01:12.533 ******** 2020-06-03T18:42:33.576 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-06-03T18:42:33.578 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-06-03T18:42:33.617 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.048) 0:01:12.582 ******** 2020-06-03T18:42:33.676 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-06-03T18:42:33.678 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-06-03T18:42:33.729 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.112) 0:01:12.695 ******** 2020-06-03T18:42:33.747 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-06-03T18:42:33.749 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-06-03T18:42:33.791 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.061) 0:01:12.757 ******** 2020-06-03T18:42:33.800 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-06-03T18:42:33.802 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-03T18:42:33.837 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:33 +0000 (0:00:00.045) 0:01:12.802 ******** 2020-06-03T18:42:33.989 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-03T18:42:34.045 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.208) 0:01:13.011 ******** 2020-06-03T18:42:34.201 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-06-03T18:42:34.204 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2020-06-03T18:42:34.237 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.191) 0:01:13.202 ******** 2020-06-03T18:42:34.256 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-06-03T18:42:34.258 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-06-03T18:42:34.294 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.057) 0:01:13.260 ******** 2020-06-03T18:42:34.304 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-06-03T18:42:34.306 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:34.342 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.047) 0:01:13.307 ******** 2020-06-03T18:42:34.349 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-06-03T18:42:34.350 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi061"}, "changed": false} 2020-06-03T18:42:34.387 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.046) 0:01:13.353 ******** 2020-06-03T18:42:34.397 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-06-03T18:42:34.398 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:34.434 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.046) 0:01:13.399 ******** 2020-06-03T18:42:34.598 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-06-03T18:42:34.599 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-06-03T18:42:34.633 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:34 +0000 (0:00:00.199) 0:01:13.598 ******** 2020-06-03T18:42:35.062 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-06-03T18:42:35.101 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:35 +0000 (0:00:00.468) 0:01:14.067 ******** 2020-06-03T18:42:35.142 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-06-03T18:42:35.144 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi112"}, "changed": false} 2020-06-03T18:42:35.180 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:35 +0000 (0:00:00.078) 0:01:14.145 ******** 2020-06-03T18:42:35.368 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi061.front.sepia.ceph.com", "ansible_hostname": "smithi061", "ansible_nodename": "smithi061"}, "changed": true, "name": "smithi061"} 2020-06-03T18:42:35.406 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:35 +0000 (0:00:00.226) 0:01:14.371 ******** 2020-06-03T18:42:35.675 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi112.front.sepia.ceph.com", "ansible_hostname": "smithi112", "ansible_nodename": "smithi112"}, "changed": true, "name": "smithi112"} 2020-06-03T18:42:35.715 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:35 +0000 (0:00:00.309) 0:01:14.681 ******** 2020-06-03T18:42:35.781 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-06-03T18:42:35.783 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-06-03T18:42:35.824 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:35 +0000 (0:00:00.109) 0:01:14.790 ******** 2020-06-03T18:42:36.076 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-06-03T18:42:36.114 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:36 +0000 (0:00:00.289) 0:01:15.080 ******** 2020-06-03T18:42:36.175 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-06-03T18:42:36.176 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-06-03T18:42:36.213 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:36 +0000 (0:00:00.098) 0:01:15.178 ******** 2020-06-03T18:42:36.452 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-06-03T18:42:36.495 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:36 +0000 (0:00:00.282) 0:01:15.461 ******** 2020-06-03T18:42:36.869 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-06-03T18:42:36.870 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-06-03T18:42:36.915 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:36 +0000 (0:00:00.420) 0:01:15.881 ******** 2020-06-03T18:42:36.966 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-06-03T18:42:36.967 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.012 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.096) 0:01:15.977 ******** 2020-06-03T18:42:37.036 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-06-03T18:42:37.038 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-06-03T18:42:37.079 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.067) 0:01:16.044 ******** 2020-06-03T18:42:37.088 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-03T18:42:37.090 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.125 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.046) 0:01:16.091 ******** 2020-06-03T18:42:37.135 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-06-03T18:42:37.136 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.172 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.138 ******** 2020-06-03T18:42:37.181 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-03T18:42:37.183 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.219 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.185 ******** 2020-06-03T18:42:37.229 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.269 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.049) 0:01:16.235 ******** 2020-06-03T18:42:37.276 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-03T18:42:37.278 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.314 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.045) 0:01:16.280 ******** 2020-06-03T18:42:37.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-03T18:42:37.325 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.361 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.046) 0:01:16.326 ******** 2020-06-03T18:42:37.372 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-03T18:42:37.374 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.411 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.049) 0:01:16.376 ******** 2020-06-03T18:42:37.420 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-03T18:42:37.422 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.458 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.424 ******** 2020-06-03T18:42:37.468 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-03T18:42:37.469 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.506 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.472 ******** 2020-06-03T18:42:37.515 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-03T18:42:37.517 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.553 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.046) 0:01:16.518 ******** 2020-06-03T18:42:37.562 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-03T18:42:37.582 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.601 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.566 ******** 2020-06-03T18:42:37.613 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-03T18:42:37.615 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.651 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.050) 0:01:16.617 ******** 2020-06-03T18:42:37.661 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-06-03T18:42:37.663 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.698 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.046) 0:01:16.663 ******** 2020-06-03T18:42:37.706 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-03T18:42:37.708 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.743 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.045) 0:01:16.709 ******** 2020-06-03T18:42:37.752 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-06-03T18:42:37.791 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.756 ******** 2020-06-03T18:42:37.800 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-06-03T18:42:37.802 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:37.839 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.804 ******** 2020-06-03T18:42:37.849 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-06-03T18:42:37.886 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.852 ******** 2020-06-03T18:42:37.894 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-06-03T18:42:37.931 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.044) 0:01:16.896 ******** 2020-06-03T18:42:37.940 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-06-03T18:42:37.978 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:37 +0000 (0:00:00.047) 0:01:16.943 ******** 2020-06-03T18:42:37.987 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-06-03T18:42:38.026 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.047) 0:01:16.991 ******** 2020-06-03T18:42:38.033 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-06-03T18:42:38.035 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.071 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.045) 0:01:17.037 ******** 2020-06-03T18:42:38.079 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-06-03T18:42:38.117 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.045) 0:01:17.082 ******** 2020-06-03T18:42:38.124 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-06-03T18:42:38.126 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.163 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.045) 0:01:17.128 ******** 2020-06-03T18:42:38.170 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-06-03T18:42:38.172 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.207 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.044) 0:01:17.173 ******** 2020-06-03T18:42:38.215 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-06-03T18:42:38.217 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.254 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.046) 0:01:17.219 ******** 2020-06-03T18:42:38.261 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-06-03T18:42:38.263 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.300 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.045) 0:01:17.265 ******** 2020-06-03T18:42:38.310 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:42:38.311 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.348 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.048) 0:01:17.314 ******** 2020-06-03T18:42:38.357 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-06-03T18:42:38.359 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.399 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.051) 0:01:17.365 ******** 2020-06-03T18:42:38.407 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-06-03T18:42:38.409 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.453 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.051) 0:01:17.416 ******** 2020-06-03T18:42:38.461 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:42:38.462 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.500 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.049) 0:01:17.465 ******** 2020-06-03T18:42:38.509 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-06-03T18:42:38.511 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.549 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.048) 0:01:17.514 ******** 2020-06-03T18:42:38.557 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-06-03T18:42:38.559 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.597 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.048) 0:01:17.562 ******** 2020-06-03T18:42:38.606 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-06-03T18:42:38.608 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.644 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.047) 0:01:17.610 ******** 2020-06-03T18:42:38.655 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-06-03T18:42:38.656 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.694 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.049) 0:01:17.659 ******** 2020-06-03T18:42:38.703 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-06-03T18:42:38.705 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.741 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.047) 0:01:17.706 ******** 2020-06-03T18:42:38.750 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-06-03T18:42:38.751 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.790 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.048) 0:01:17.755 ******** 2020-06-03T18:42:38.798 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-03T18:42:38.800 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.837 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.047) 0:01:17.803 ******** 2020-06-03T18:42:38.846 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-06-03T18:42:38.848 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.884 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.047) 0:01:17.850 ******** 2020-06-03T18:42:38.892 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-03T18:42:38.894 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.931 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.046) 0:01:17.896 ******** 2020-06-03T18:42:38.939 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-03T18:42:38.941 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:38.980 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:38 +0000 (0:00:00.048) 0:01:17.945 ******** 2020-06-03T18:42:38.988 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:42:38.990 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.026 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.046) 0:01:17.991 ******** 2020-06-03T18:42:39.034 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-03T18:42:39.036 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.072 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.046) 0:01:18.038 ******** 2020-06-03T18:42:39.082 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-06-03T18:42:39.083 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.121 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.048) 0:01:18.086 ******** 2020-06-03T18:42:39.129 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:42:39.131 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.167 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.046) 0:01:18.133 ******** 2020-06-03T18:42:39.177 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-03T18:42:39.179 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.216 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.048) 0:01:18.181 ******** 2020-06-03T18:42:39.224 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-06-03T18:42:39.226 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.262 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.046) 0:01:18.228 ******** 2020-06-03T18:42:39.272 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:42:39.274 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.314 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.051) 0:01:18.279 ******** 2020-06-03T18:42:39.323 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-03T18:42:39.325 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.369 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.055) 0:01:18.334 ******** 2020-06-03T18:42:39.379 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-06-03T18:42:39.381 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.420 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.051) 0:01:18.386 ******** 2020-06-03T18:42:39.430 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:42:39.432 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.468 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.047) 0:01:18.433 ******** 2020-06-03T18:42:39.478 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-06-03T18:42:39.480 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.521 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.053) 0:01:18.487 ******** 2020-06-03T18:42:39.531 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-06-03T18:42:39.533 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.577 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.055) 0:01:18.542 ******** 2020-06-03T18:42:39.586 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-06-03T18:42:39.588 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.625 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.048) 0:01:18.591 ******** 2020-06-03T18:42:39.633 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-06-03T18:42:39.635 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.679 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.053) 0:01:18.644 ******** 2020-06-03T18:42:39.688 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-06-03T18:42:39.690 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.728 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.049) 0:01:18.694 ******** 2020-06-03T18:42:39.737 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-06-03T18:42:39.788 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.794 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.065) 0:01:18.759 ******** 2020-06-03T18:42:39.803 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-06-03T18:42:39.805 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.843 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.049) 0:01:18.809 ******** 2020-06-03T18:42:39.855 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-06-03T18:42:39.857 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.894 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.050) 0:01:18.859 ******** 2020-06-03T18:42:39.904 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-06-03T18:42:39.906 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:39.946 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.052) 0:01:18.912 ******** 2020-06-03T18:42:39.956 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-06-03T18:42:39.958 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.001 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:39 +0000 (0:00:00.054) 0:01:18.967 ******** 2020-06-03T18:42:40.011 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-06-03T18:42:40.013 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.049 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.047) 0:01:19.015 ******** 2020-06-03T18:42:40.065 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-06-03T18:42:40.066 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.105 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.056) 0:01:19.071 ******** 2020-06-03T18:42:40.115 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-06-03T18:42:40.117 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.154 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.048) 0:01:19.120 ******** 2020-06-03T18:42:40.162 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-06-03T18:42:40.164 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.212 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.057) 0:01:19.178 ******** 2020-06-03T18:42:40.245 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-06-03T18:42:40.247 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.287 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.074) 0:01:19.252 ******** 2020-06-03T18:42:40.309 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-06-03T18:42:40.311 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.006078", "end": "2020-06-03 18:42:40.008362", "rc": 0, "start": "2020-06-03 18:42:40.002284", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:40.358 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.071) 0:01:19.324 ******** 2020-06-03T18:42:40.405 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-06-03T18:42:40.458 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.099) 0:01:19.423 ******** 2020-06-03T18:42:40.475 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-06-03T18:42:40.477 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005154", "end": "2020-06-03 18:42:40.051031", "rc": 0, "start": "2020-06-03 18:42:40.045877", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:40.519 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.061) 0:01:19.485 ******** 2020-06-03T18:42:40.553 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-06-03T18:42:40.593 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.073) 0:01:19.558 ******** 2020-06-03T18:42:40.811 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-06-03T18:42:40.813 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-06-03T18:42:40.855 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.262) 0:01:19.821 ******** 2020-06-03T18:42:40.897 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-06-03T18:42:40.900 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:40.937 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.081) 0:01:19.902 ******** 2020-06-03T18:42:40.949 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-06-03T18:42:40.951 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-06-03T18:42:40.996 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:40 +0000 (0:00:00.059) 0:01:19.962 ******** 2020-06-03T18:42:41.044 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:42:41.077 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:41 +0000 (0:00:00.081) 0:01:20.043 ******** 2020-06-03T18:42:42.297 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-06-03T18:42:42.309 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-06-03T18:42:42.364 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-06-03T18:42:43.251 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-06-03T18:42:43.289 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:43 +0000 (0:00:02.211) 0:01:22.254 ******** 2020-06-03T18:42:43.387 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-06-03T18:42:43.441 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:43 +0000 (0:00:00.152) 0:01:22.407 ******** 2020-06-03T18:42:44.211 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-06-03T18:42:44.213 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:42:44.249 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.807) 0:01:23.214 ******** 2020-06-03T18:42:44.278 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-06-03T18:42:44.313 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.064) 0:01:23.279 ******** 2020-06-03T18:42:44.405 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-06-03T18:42:44.407 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:42:44.455 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.142) 0:01:23.421 ******** 2020-06-03T18:42:44.488 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-06-03T18:42:44.490 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034491", "end": "2020-06-03 18:42:44.104958", "rc": 0, "start": "2020-06-03 18:42:44.070467", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:44.525 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.069) 0:01:23.491 ******** 2020-06-03T18:42:44.562 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.036) 0:01:23.527 ******** 2020-06-03T18:42:44.706 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035749", "end": "2020-06-03 18:42:44.413140", "rc": 0, "start": "2020-06-03 18:42:44.377391", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:42:44.745 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:44 +0000 (0:00:00.183) 0:01:23.710 ******** 2020-06-03T18:42:47.261 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-06-03T18:42:47.263 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:42:47.297 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:47 +0000 (0:00:02.552) 0:01:26.262 ******** 2020-06-03T18:42:47.694 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:42:47.736 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:47 +0000 (0:00:00.438) 0:01:26.701 ******** 2020-06-03T18:42:57.120 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-06-03T18:42:57.122 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137515 files and directorie 2020-06-03T18:42:57.124 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-06-03T18:42:57.125 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137515 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:42:57.158 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:57 +0000 (0:00:09.421) 0:01:36.123 ******** 2020-06-03T18:42:58.038 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137515 files and directorie 2020-06-03T18:42:58.040 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-06-03T18:42:58.042 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137515 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:42:58.072 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:42:58 +0000 (0:00:00.913) 0:01:37.037 ******** 2020-06-03T18:43:06.946 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:43:06.948 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apt-utils libapt-pkg5.0\nSuggested packages:\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apt apt-utils libapt-pkg5.0\n3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 2214 kB of archives.\nAfter this operation, 1024 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1201 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]\nFetched 2214 kB in 1s (2620 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database . 2020-06-03T18:43:06.950 INFO:teuthology.task.ansible.out:.. 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading da 2020-06-03T18:43:06.952 INFO:teuthology.task.ansible.out:tabase ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt-utils (1.6.12ubuntu0.1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apt-utils libapt-pkg5.0", "Suggested packages:", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apt apt-utils libapt-pkg5.0", "3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.", "Need to get 2214 kB of archives.", "After this operation, 1024 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1 2020-06-03T18:43:06.953 INFO:teuthology.task.ansible.out:201 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]", "Fetched 2214 kB in 1s (2620 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database 2020-06-03T18:43:06.955 INFO:teuthology.task.ansible.out: ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt-utils (1.6.12ubuntu0.1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:43:06.985 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:06 +0000 (0:00:08.912) 0:01:45.950 ******** 2020-06-03T18:43:07.014 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-06-03T18:43:07.016 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:07.052 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:07 +0000 (0:00:00.066) 0:01:46.017 ******** 2020-06-03T18:43:08.272 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:43:08.273 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apt-utils libapt-pkg5.0\nSuggested packages:\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apt apt-utils libapt-pkg5.0\n3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 2214 kB of archives.\nAfter this operation, 1024 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1201 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]\nFetched 2214 kB in 1s (2550 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database . 2020-06-03T18:43:08.274 INFO:teuthology.task.ansible.out:.. 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading da 2020-06-03T18:43:08.276 INFO:teuthology.task.ansible.out:tabase ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt-utils (1.6.12ubuntu0.1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apt-utils libapt-pkg5.0", "Suggested packages:", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apt apt-utils libapt-pkg5.0", "3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.", "Need to get 2214 kB of archives.", "After this operation, 1024 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1 2020-06-03T18:43:08.278 INFO:teuthology.task.ansible.out:201 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]", "Fetched 2214 kB in 1s (2550 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database 2020-06-03T18:43:08.279 INFO:teuthology.task.ansible.out: ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt-utils (1.6.12ubuntu0.1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:43:08.311 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:08 +0000 (0:00:01.259) 0:01:47.277 ******** 2020-06-03T18:43:08.353 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-06-03T18:43:08.355 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:08.391 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:08 +0000 (0:00:00.079) 0:01:47.356 ******** 2020-06-03T18:43:16.393 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:43:16.394 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]\nFetched 10.4 MB in 1s (8920 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-06-03T18:43:16.396 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-06-03T18:43:16.397 INFO:teuthology.task.ansible.out:u1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]", "Fetched 10.4 MB in 1s (8920 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-06-03T18:43:16.399 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:43:16.430 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:16 +0000 (0:00:08.039) 0:01:55.396 ******** 2020-06-03T18:43:17.360 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-06-03T18:43:17.362 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:43:17.397 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:17 +0000 (0:00:00.966) 0:01:56.362 ******** 2020-06-03T18:43:17.532 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-06-03T18:43:17.534 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-06-03T18:43:17.571 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:17 +0000 (0:00:00.172) 0:01:56.535 ******** 2020-06-03T18:43:17.599 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-06-03T18:43:17.601 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:17.634 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:17 +0000 (0:00:00.064) 0:01:56.599 ******** 2020-06-03T18:43:17.776 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-06-03T18:43:17.777 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-06-03T18:43:17.811 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:17 +0000 (0:00:00.177) 0:01:56.777 ******** 2020-06-03T18:43:17.936 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:43:17.938 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]\nFetched 10.4 MB in 1s (9608 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-06-03T18:43:17.939 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-06-03T18:43:17.941 INFO:teuthology.task.ansible.out:u1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]", "Fetched 10.4 MB in 1s (9608 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-06-03T18:43:17.943 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-03T18:43:17.976 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:17 +0000 (0:00:00.164) 0:01:56.941 ******** 2020-06-03T18:43:18.159 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-06-03T18:43:18.161 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-06-03T18:43:18.195 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:18 +0000 (0:00:00.219) 0:01:57.161 ******** 2020-06-03T18:43:18.359 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-06-03T18:43:18.361 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-03T18:43:18.395 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:18 +0000 (0:00:00.199) 0:01:57.361 ******** 2020-06-03T18:43:18.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-06-03T18:43:18.570 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:18.603 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:18 +0000 (0:00:00.208) 0:01:57.569 ******** 2020-06-03T18:43:18.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-06-03T18:43:18.862 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:43:18.896 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:18 +0000 (0:00:00.292) 0:01:57.861 ******** 2020-06-03T18:43:18.923 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-06-03T18:43:18.924 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ActiveEnterTimestampMonotonic": "19161394", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-tmpfiles-setup.service -.mount system.slice sysinit.target systemd-journald.socket network.target nss-lookup.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:40:53 UTC", "AssertTimestampMonotonic": "18608342", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ConditionTimestampMonotonic": "18608342", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-06-03T18:43:18.926 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1241", "ExecMainStartTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ExecMainStartTimestampMonotonic": "19161376", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:40:53 UTC", "InactiveExitTimestampMonotonic": "18609031", "InvocationID": "9342473068f449d3a1c509387e7b7bd2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-06-03T18:43:18.927 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1241", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-06-03T18:43:18.929 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:40:53 UTC", "StateChangeTimestampMonotonic": "19161394", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:40:53 UTC", "WatchdogTimestampMonotonic": "19161392", "WatchdogUSec": "0"}} 2020-06-03T18:43:18.962 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:18 +0000 (0:00:00.066) 0:01:57.928 ******** 2020-06-03T18:43:19.003 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-03T18:43:19.004 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.039 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.076) 0:01:58.004 ******** 2020-06-03T18:43:19.046 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-06-03T18:43:19.048 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-06-03T18:43:19.085 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.046) 0:01:58.050 ******** 2020-06-03T18:43:19.093 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-03T18:43:19.095 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.138 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.052) 0:01:58.103 ******** 2020-06-03T18:43:19.147 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-06-03T18:43:19.149 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.185 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.047) 0:01:58.150 ******** 2020-06-03T18:43:19.193 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-03T18:43:19.194 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.231 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.046) 0:01:58.197 ******** 2020-06-03T18:43:19.280 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-03T18:43:19.282 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.322 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.090) 0:01:58.287 ******** 2020-06-03T18:43:19.339 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-06-03T18:43:19.340 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-06-03T18:43:19.375 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.053) 0:01:58.341 ******** 2020-06-03T18:43:19.384 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-03T18:43:19.386 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.422 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.046) 0:01:58.387 ******** 2020-06-03T18:43:19.461 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-03T18:43:19.463 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.496 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.074) 0:01:58.462 ******** 2020-06-03T18:43:19.525 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-03T18:43:19.527 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.561 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.064) 0:01:58.526 ******** 2020-06-03T18:43:19.601 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-03T18:43:19.602 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.636 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.075) 0:01:58.602 ******** 2020-06-03T18:43:19.676 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:19.678 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.712 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.075) 0:01:58.677 ******** 2020-06-03T18:43:19.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-06-03T18:43:19.760 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.047) 0:01:58.725 ******** 2020-06-03T18:43:19.770 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-03T18:43:19.772 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.808 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.047) 0:01:58.773 ******** 2020-06-03T18:43:19.838 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:43:19.840 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.878 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.069) 0:01:58.843 ******** 2020-06-03T18:43:19.919 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:43:19.921 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:19.964 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:19 +0000 (0:00:00.086) 0:01:58.929 ******** 2020-06-03T18:43:19.974 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-06-03T18:43:19.976 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-03T18:43:20.013 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:58.979 ******** 2020-06-03T18:43:20.024 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-03T18:43:20.026 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.063 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:59.028 ******** 2020-06-03T18:43:20.105 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:20.107 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.149 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.082) 0:01:59.111 ******** 2020-06-03T18:43:20.175 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-06-03T18:43:20.177 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:20.213 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.067) 0:01:59.179 ******** 2020-06-03T18:43:20.223 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-03T18:43:20.225 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.266 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.048) 0:01:59.227 ******** 2020-06-03T18:43:20.324 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-03T18:43:20.325 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.372 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.110) 0:01:59.338 ******** 2020-06-03T18:43:20.414 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-03T18:43:20.416 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.450 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.077) 0:01:59.415 ******** 2020-06-03T18:43:20.491 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-03T18:43:20.493 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.528 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.077) 0:01:59.493 ******** 2020-06-03T18:43:20.552 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-06-03T18:43:20.554 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ActiveEnterTimestampMonotonic": "18915569", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target sysinit.target system.slice -.mount nss-lookup.target remote-fs.target network.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:41:04 UTC", "AssertTimestampMonotonic": "18396505", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ConditionTimestampMonotonic": "18396505", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-06-03T18:43:20.556 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1398", "ExecMainStartTimestamp": "Wed 2020-06-03 18:41:04 UTC", "ExecMainStartTimestampMonotonic": "18915559", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:41:04 UTC", "InactiveExitTimestampMonotonic": "18397200", "InvocationID": "0c4115185b7b49959c745f9d1b830f0b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-06-03T18:43:20.560 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1398", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-06-03T18:43:20.562 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:41:04 UTC", "StateChangeTimestampMonotonic": "18915569", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:41:04 UTC", "WatchdogTimestampMonotonic": "18915568", "WatchdogUSec": "0"}} 2020-06-03T18:43:20.592 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.063) 0:01:59.557 ******** 2020-06-03T18:43:20.600 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-03T18:43:20.602 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.642 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.047) 0:01:59.605 ******** 2020-06-03T18:43:20.649 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-03T18:43:20.650 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.689 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:59.655 ******** 2020-06-03T18:43:20.698 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:20.699 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.737 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.047) 0:01:59.702 ******** 2020-06-03T18:43:20.746 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.786 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:59.751 ******** 2020-06-03T18:43:20.796 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-03T18:43:20.798 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.835 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.048) 0:01:59.800 ******** 2020-06-03T18:43:20.844 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-03T18:43:20.847 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.884 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:59.850 ******** 2020-06-03T18:43:20.895 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-03T18:43:20.897 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.934 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.050) 0:01:59.900 ******** 2020-06-03T18:43:20.944 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-03T18:43:20.946 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:20.984 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:20 +0000 (0:00:00.049) 0:01:59.950 ******** 2020-06-03T18:43:20.993 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:20.995 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.031 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.046) 0:01:59.997 ******** 2020-06-03T18:43:21.041 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-03T18:43:21.042 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.081 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.048) 0:02:00.046 ******** 2020-06-03T18:43:21.091 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.129 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.048) 0:02:00.094 ******** 2020-06-03T18:43:21.139 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-03T18:43:21.141 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.181 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.051) 0:02:00.146 ******** 2020-06-03T18:43:21.192 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-03T18:43:21.193 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.231 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.050) 0:02:00.197 ******** 2020-06-03T18:43:21.242 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-03T18:43:21.243 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.283 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.051) 0:02:00.249 ******** 2020-06-03T18:43:21.294 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-06-03T18:43:21.296 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.342 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.058) 0:02:00.307 ******** 2020-06-03T18:43:21.352 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-03T18:43:21.354 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.394 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.052) 0:02:00.359 ******** 2020-06-03T18:43:21.423 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:21.425 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.460 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.065) 0:02:00.425 ******** 2020-06-03T18:43:21.490 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-03T18:43:21.492 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-06-03T18:43:21.530 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.069) 0:02:00.495 ******** 2020-06-03T18:43:21.567 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.037) 0:02:00.532 ******** 2020-06-03T18:43:21.575 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-03T18:43:21.577 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.620 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.052) 0:02:00.585 ******** 2020-06-03T18:43:21.650 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-03T18:43:21.652 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.689 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.069) 0:02:00.655 ******** 2020-06-03T18:43:21.719 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-03T18:43:21.721 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.758 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.069) 0:02:00.724 ******** 2020-06-03T18:43:21.788 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:21.790 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.825 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.066) 0:02:00.791 ******** 2020-06-03T18:43:21.854 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-03T18:43:21.856 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:21.894 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.068) 0:02:00.859 ******** 2020-06-03T18:43:21.912 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-03T18:43:21.914 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-06-03T18:43:21.951 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.056) 0:02:00.916 ******** 2020-06-03T18:43:21.960 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-03T18:43:21.962 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.001 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:21 +0000 (0:00:00.050) 0:02:00.966 ******** 2020-06-03T18:43:22.011 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-03T18:43:22.013 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.051 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.049) 0:02:01.016 ******** 2020-06-03T18:43:22.059 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-03T18:43:22.061 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.099 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.048) 0:02:01.065 ******** 2020-06-03T18:43:22.139 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-03T18:43:22.141 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.179 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.080) 0:02:01.145 ******** 2020-06-03T18:43:22.220 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-03T18:43:22.222 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.257 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.077) 0:02:01.223 ******** 2020-06-03T18:43:22.287 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:22.288 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.324 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.067) 0:02:01.290 ******** 2020-06-03T18:43:22.355 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-03T18:43:22.357 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.391 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.066) 0:02:01.357 ******** 2020-06-03T18:43:22.411 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-06-03T18:43:22.412 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-06-03T18:43:22.447 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.055) 0:02:01.412 ******** 2020-06-03T18:43:22.456 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-03T18:43:22.458 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.496 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.048) 0:02:01.461 ******** 2020-06-03T18:43:22.524 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-03T18:43:22.526 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.562 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.066) 0:02:01.527 ******** 2020-06-03T18:43:22.611 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-03T18:43:22.612 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.647 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.085) 0:02:01.613 ******** 2020-06-03T18:43:22.678 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.718 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.070) 0:02:01.684 ******** 2020-06-03T18:43:22.758 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-06-03T18:43:22.760 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:22.802 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.083) 0:02:01.767 ******** 2020-06-03T18:43:22.821 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-06-03T18:43:22.823 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-06-03T18:43:22.859 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.057) 0:02:01.825 ******** 2020-06-03T18:43:22.942 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-06-03T18:43:22.943 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-06-03T18:43:22.979 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:22 +0000 (0:00:00.119) 0:02:01.945 ******** 2020-06-03T18:43:23.222 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-03T18:43:23.224 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-06-03T18:43:23.260 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.280) 0:02:02.225 ******** 2020-06-03T18:43:23.331 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-03T18:43:23.334 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-06-03T18:43:23.369 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.109) 0:02:02.335 ******** 2020-06-03T18:43:23.399 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-03T18:43:23.401 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.434 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.065) 0:02:02.400 ******** 2020-06-03T18:43:23.443 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-03T18:43:23.444 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:23.486 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.051) 0:02:02.451 ******** 2020-06-03T18:43:23.525 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-06-03T18:43:23.527 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.560 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.073) 0:02:02.525 ******** 2020-06-03T18:43:23.598 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-03T18:43:23.600 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.634 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.073) 0:02:02.599 ******** 2020-06-03T18:43:23.662 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-06-03T18:43:23.664 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.697 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.063) 0:02:02.663 ******** 2020-06-03T18:43:23.725 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-06-03T18:43:23.727 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.761 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.063) 0:02:02.726 ******** 2020-06-03T18:43:23.801 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-06-03T18:43:23.803 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.837 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.075) 0:02:02.802 ******** 2020-06-03T18:43:23.847 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-06-03T18:43:23.885 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.047) 0:02:02.849 ******** 2020-06-03T18:43:23.893 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:23.895 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-03T18:43:23.931 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:23 +0000 (0:00:00.047) 0:02:02.897 ******** 2020-06-03T18:43:23.980 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-06-03T18:43:23.982 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.991 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:23.993 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.003 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.013 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.015 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.036 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-03T18:43:24.047 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.049 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.058 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.068 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.071 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.080 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.091 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.092 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.133 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.201) 0:02:03.098 ******** 2020-06-03T18:43:24.244 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-06-03T18:43:24.246 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-06-03T18:43:24.285 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.152) 0:02:03.251 ******** 2020-06-03T18:43:24.370 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-06-03T18:43:24.371 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004235", "end": "2020-06-03 18:43:24.341450", "rc": 0, "start": "2020-06-03 18:43:24.337215", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-03T18:43:24.413 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.127) 0:02:03.378 ******** 2020-06-03T18:43:24.456 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-06-03T18:43:24.457 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.500 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.087) 0:02:03.466 ******** 2020-06-03T18:43:24.540 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-06-03T18:43:24.541 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.580 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.080) 0:02:03.546 ******** 2020-06-03T18:43:24.622 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-03T18:43:24.623 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.662 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.081) 0:02:03.627 ******** 2020-06-03T18:43:24.671 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-03T18:43:24.672 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-06-03T18:43:24.718 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.056) 0:02:03.684 ******** 2020-06-03T18:43:24.874 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-03T18:43:24.875 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:24.913 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.195) 0:02:03.879 ******** 2020-06-03T18:43:24.946 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-06-03T18:43:24.947 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:24.984 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:24 +0000 (0:00:00.070) 0:02:03.950 ******** 2020-06-03T18:43:25.015 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-03T18:43:25.016 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.054 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.069) 0:02:04.019 ******** 2020-06-03T18:43:25.084 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-06-03T18:43:25.085 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.122 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.068) 0:02:04.088 ******** 2020-06-03T18:43:25.152 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-06-03T18:43:25.153 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.190 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.067) 0:02:04.155 ******** 2020-06-03T18:43:25.231 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-06-03T18:43:25.232 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.270 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.079) 0:02:04.235 ******** 2020-06-03T18:43:25.311 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:25.312 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-03T18:43:25.356 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.086) 0:02:04.321 ******** 2020-06-03T18:43:25.417 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-06-03T18:43:25.418 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.420 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.422 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.431 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.443 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.444 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.455 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-03T18:43:25.466 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.467 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.479 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.480 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.491 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.492 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.504 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.505 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.548 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.192) 0:02:04.513 ******** 2020-06-03T18:43:25.558 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-06-03T18:43:25.560 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:43:25.604 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.056) 0:02:04.570 ******** 2020-06-03T18:43:25.634 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-03T18:43:25.635 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.680 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.076) 0:02:04.646 ******** 2020-06-03T18:43:25.732 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-03T18:43:25.733 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item={'block_used': 13741036, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354072, 'size_available': 927146278912, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354072, "block_size": 4096, "block_total": 240095108, "block_used": 13741036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146278912, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}} 2020-06-03T18:43:25.774 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.093) 0:02:04.739 ******** 2020-06-03T18:43:25.782 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-06-03T18:43:25.784 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003867", "end": "2020-06-03 18:43:25.747537", "rc": 0, "start": "2020-06-03 18:43:25.743670", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-03T18:43:25.827 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.052) 0:02:04.792 ******** 2020-06-03T18:43:25.872 INFO:teuthology.task.ansible.out: 2020-06-03T18:43:25.874 INFO:teuthology.task.ansible.out:TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-06-03T18:43:25.876 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:25.878 INFO:teuthology.task.ansible.out: 2020-06-03T18:43:25.925 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.098) 0:02:04.891 ******** 2020-06-03T18:43:25.939 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-03T18:43:25.941 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010205", "end": "2020-06-03 18:43:25.561723", "rc": 0, "start": "2020-06-03 18:43:25.551518", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:25.983 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:25 +0000 (0:00:00.057) 0:02:04.949 ******** 2020-06-03T18:43:25.994 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-06-03T18:43:25.996 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:26.041 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:26 +0000 (0:00:00.058) 0:02:05.007 ******** 2020-06-03T18:43:26.053 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-03T18:43:26.055 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'block_used': 13741036, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354072, 'size_available': 927146278912, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354072, "block_size": 4096, "block_total": 240095108, "block_used": 13741036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146278912, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:26.103 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:26 +0000 (0:00:00.061) 0:02:05.068 ******** 2020-06-03T18:43:26.114 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-03T18:43:26.116 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:26.155 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:26 +0000 (0:00:00.052) 0:02:05.121 ******** 2020-06-03T18:43:26.166 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-03T18:43:26.211 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:26 +0000 (0:00:00.055) 0:02:05.176 ******** 2020-06-03T18:43:26.280 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-06-03T18:43:26.282 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:27.100 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-06-03T18:43:27.102 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:43:27.146 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.934) 0:02:06.111 ******** 2020-06-03T18:43:27.178 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-03T18:43:27.180 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:27.219 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.072) 0:02:06.184 ******** 2020-06-03T18:43:27.273 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-03T18:43:27.275 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}} 2020-06-03T18:43:27.326 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.107) 0:02:06.292 ******** 2020-06-03T18:43:27.482 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-03T18:43:27.484 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009930", "end": "2020-06-03 18:43:27.214321", "rc": 0, "start": "2020-06-03 18:43:27.204391", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:27.524 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.197) 0:02:06.489 ******** 2020-06-03T18:43:27.577 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-03T18:43:27.579 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:27.619 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.095) 0:02:06.585 ******** 2020-06-03T18:43:27.650 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-03T18:43:27.691 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:27 +0000 (0:00:00.071) 0:02:06.656 ******** 2020-06-03T18:43:27.755 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-06-03T18:43:27.757 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:28.097 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4P9GPTazOSdGxSRDNPcikIVCAnE7iT7N5'], 'uuids': ['302be343-d020-420d-9b8b-a26d962fa85a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.689200", "end": "2020-06-03 18:43:27.728384", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4P9GPTazOSdGxSRDNPcikIVCAnE7iT7N5"], "labels": [], "masters": [], "uuids": ["302be343-d020-420d-9b8b-a26d962fa85a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:26.039184", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:28.130 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:28.132 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:29.414 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d40u1B67MH87bfypbEwHquFfbMIOWop4Ff'], 'uuids': ['c5e37dd3-56bc-4403-b4a1-24a28746221b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.150930", "end": "2020-06-03 18:43:29.041298", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d40u1B67MH87bfypbEwHquFfbMIOWop4Ff"], "labels": [], "masters": [], "uuids": ["c5e37dd3-56bc-4403-b4a1-24a28746221b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:27.890368", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:29.425 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS', 'wwn-0x5000c50091e2ee9d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'wwn-0x5000c50091e2ee9d-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e2ee9d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QZGS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS", "wwn-0x5000c50091e2ee9d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1", "wwn-0x5000c50091e2ee9d-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QZGS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2ee9d"}}, "skip_reason": 2020-06-03T18:43:29.427 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-03T18:43:29.621 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2H0szmR5iMmhRVdvE4LGnmvxYSwzqaIDoe'], 'uuids': ['e02b8bc8-8738-466c-b38a-509a8534b8ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.726249", "end": "2020-06-03 18:43:29.339083", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2H0szmR5iMmhRVdvE4LGnmvxYSwzqaIDoe"], "labels": [], "masters": [], "uuids": ["e02b8bc8-8738-466c-b38a-509a8534b8ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:27.612834", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:29.622 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:29.633 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:30.789 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-xC2djQ-zXjA-ZIQT-aj6U-JN8v-cUAD-lDiWaG', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN', 'nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.213667", "end": "2020-06-03 18:43:30.412358", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-xC2djQ-zXjA-ZIQT-aj6U-JN8v-cUAD-lDiWaG", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN", "nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:29.198691", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-06-03T18:43:30.791 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:30.801 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:30.812 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:30.885 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HojRJdCTbEdz4VD0jY0AEj2hdryEDUdV7'], 'uuids': ['6a7c86ad-e6ee-49ed-b9c6-7d4073796d1a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.126699", "end": "2020-06-03 18:43:30.610069", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HojRJdCTbEdz4VD0jY0AEj2hdryEDUdV7"], "labels": [], "masters": [], "uuids": ["6a7c86ad-e6ee-49ed-b9c6-7d4073796d1a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:29.483370", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:30.897 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8', 'wwn-0x5000c50087b8cd81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1', 'wwn-0x5000c50087b8cd81-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8cd81', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2W8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8", "wwn-0x5000c50087b8cd81"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1", "wwn-0x5000c50087b8cd81-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2W8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8cd81"}}, "skip_reason": 2020-06-03T18:43:30.899 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-03T18:43:32.126 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4NrigXLMgmVs7YlCegoWaBIEIqg1QC1LU'], 'uuids': ['eb978243-223a-4493-8c7f-8dfc1799660f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.182802", "end": "2020-06-03 18:43:31.751033", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4NrigXLMgmVs7YlCegoWaBIEIqg1QC1LU"], "labels": [], "masters": [], "uuids": ["eb978243-223a-4493-8c7f-8dfc1799660f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:30.568231", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:32.143 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:32.159 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:32.202 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ld7T9Z-2Plu-y43P-Xv5R-Sa2Q-FAjr-e0Jc5y', 'nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN', 'nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.183870", "end": "2020-06-03 18:43:31.927035", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ld7T9Z-2Plu-y43P-Xv5R-Sa2Q-FAjr-e0Jc5y", "nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN", "nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:30.743165", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-06-03T18:43:32.204 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:32.217 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:32.239 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:33.431 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4ZuWSYrkF00Uqthnw9S62O5NmPA5iJwxQ'], 'uuids': ['79799bed-c45c-4b4a-a381-da7aafcb0dfe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.165582", "end": "2020-06-03 18:43:33.062635", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4ZuWSYrkF00Uqthnw9S62O5NmPA5iJwxQ"], "labels": [], "masters": [], "uuids": ["79799bed-c45c-4b4a-a381-da7aafcb0dfe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:31.897053", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:33.443 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:33.485 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HqkMm7qvbNtHzh6y4h9qMvKJoQgHg9v2Z'], 'uuids': ['b1adb7cb-24ac-4d22-995a-3f4c4a1aaa26']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.120601", "end": "2020-06-03 18:43:33.220319", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HqkMm7qvbNtHzh6y4h9qMvKJoQgHg9v2Z"], "labels": [], "masters": [], "uuids": ["b1adb7cb-24ac-4d22-995a-3f4c4a1aaa26"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:32.099718", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:33.496 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:33.508 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:34.753 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4gQBdt1hiAzvL0DygGR16XHop5CdllgdX'], 'uuids': ['636cf496-e1f0-4a44-8a2a-09fc320a9f7c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.173413", "end": "2020-06-03 18:43:34.381662", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4gQBdt1hiAzvL0DygGR16XHop5CdllgdX"], "labels": [], "masters": [], "uuids": ["636cf496-e1f0-4a44-8a2a-09fc320a9f7c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:33.208249", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:34.764 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HO9Uh7w4n9ijkPelLt92yy2JpOkXeXnqp'], 'uuids': ['c4009b40-c05a-47bd-a8a8-6b5f0b6812bc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.125082", "end": "2020-06-03 18:43:34.487314", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HO9Uh7w4n9ijkPelLt92yy2JpOkXeXnqp"], "labels": [], "masters": [], "uuids": ["c4009b40-c05a-47bd-a8a8-6b5f0b6812bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:33.362232", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:34.765 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:34.815 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:34 +0000 (0:00:07.124) 0:02:13.781 ******** 2020-06-03T18:43:34.899 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:34.901 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.189 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4P9GPTazOSdGxSRDNPcikIVCAnE7iT7N5'], 'uuids': ['302be343-d020-420d-9b8b-a26d962fa85a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.169189", "end": "2020-06-03 18:43:34.828712", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4P9GPTazOSdGxSRDNPcikIVCAnE7iT7N5"], "labels": [], "masters": [], "uuids": ["302be343-d020-420d-9b8b-a26d962fa85a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:34.659523", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15391 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15391 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:35.200 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.211 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.491 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d40u1B67MH87bfypbEwHquFfbMIOWop4Ff'], 'uuids': ['c5e37dd3-56bc-4403-b4a1-24a28746221b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.155769", "end": "2020-06-03 18:43:35.115789", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d40u1B67MH87bfypbEwHquFfbMIOWop4Ff"], "labels": [], "masters": [], "uuids": ["c5e37dd3-56bc-4403-b4a1-24a28746221b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:34.960020", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15116 s, 763 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15116 s, 763 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:35.502 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS', 'wwn-0x5000c50091e2ee9d'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1', 'wwn-0x5000c50091e2ee9d-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e2ee9d', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QZGS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS", "wwn-0x5000c50091e2ee9d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QZGS-part1", "wwn-0x5000c50091e2ee9d-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QZGS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2ee9d"}}, "skip_reason": 2020-06-03T18:43:35.504 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-03T18:43:35.680 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-xC2djQ-zXjA-ZIQT-aj6U-JN8v-cUAD-lDiWaG', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN', 'nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052817", "end": "2020-06-03 18:43:35.308379", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-xC2djQ-zXjA-ZIQT-aj6U-JN8v-cUAD-lDiWaG", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NV400BGN", "nvme-nvme.8086-504846543632303430304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:35.255562", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0480354 s, 2.4 GB/s", "std 2020-06-03T18:43:35.682 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0480354 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:35.691 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.703 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.971 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4NrigXLMgmVs7YlCegoWaBIEIqg1QC1LU'], 'uuids': ['eb978243-223a-4493-8c7f-8dfc1799660f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.148915", "end": "2020-06-03 18:43:35.605726", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4NrigXLMgmVs7YlCegoWaBIEIqg1QC1LU"], "labels": [], "masters": [], "uuids": ["eb978243-223a-4493-8c7f-8dfc1799660f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:35.456811", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145719 s, 792 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145719 s, 792 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:35.982 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:35.994 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.005 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-06-03T18:43:36.006 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HZZhytcd4Hmycfxfk1YuwFQCrXz0dpC9l'], 'uuids': ['67df8254-05c7-4b94-9626-517ee202d345']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.124647", "end": "2020-06-03 18:43:35.738523", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HZZhytcd4Hmycfxfk1YuwFQCrXz0dpC9l"], "labels": [], "masters": [], "uuids": ["67df8254-05c7-4b94-9626-517ee202d345"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:34.613876", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-03T18:43:36.088 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:36 +0000 (0:00:01.272) 0:02:15.054 ******** 2020-06-03T18:43:36.159 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:36.161 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.275 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4ZuWSYrkF00Uqthnw9S62O5NmPA5iJwxQ'], 'uuids': ['79799bed-c45c-4b4a-a381-da7aafcb0dfe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.154274", "end": "2020-06-03 18:43:35.900034", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4ZuWSYrkF00Uqthnw9S62O5NmPA5iJwxQ"], "labels": [], "masters": [], "uuids": ["79799bed-c45c-4b4a-a381-da7aafcb0dfe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:35.745760", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149657 s, 771 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149657 s, 771 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.290 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.452 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2H0szmR5iMmhRVdvE4LGnmvxYSwzqaIDoe'], 'uuids': ['e02b8bc8-8738-466c-b38a-509a8534b8ee']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.161668", "end": "2020-06-03 18:43:36.177033", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2H0szmR5iMmhRVdvE4LGnmvxYSwzqaIDoe"], "labels": [], "masters": [], "uuids": ["e02b8bc8-8738-466c-b38a-509a8534b8ee"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:36.015365", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144901 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144901 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.464 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.475 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.574 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4gQBdt1hiAzvL0DygGR16XHop5CdllgdX'], 'uuids': ['636cf496-e1f0-4a44-8a2a-09fc320a9f7c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155293", "end": "2020-06-03 18:43:36.201362", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-lrDTFd8bB2wl3SsGWFabUjsglJk7C1d4gQBdt1hiAzvL0DygGR16XHop5CdllgdX"], "labels": [], "masters": [], "uuids": ["636cf496-e1f0-4a44-8a2a-09fc320a9f7c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:36.046069", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150704 s, 765 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150704 s, 765 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.629 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:36 +0000 (0:00:00.541) 0:02:15.595 ******** 2020-06-03T18:43:36.774 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HojRJdCTbEdz4VD0jY0AEj2hdryEDUdV7'], 'uuids': ['6a7c86ad-e6ee-49ed-b9c6-7d4073796d1a']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159552", "end": "2020-06-03 18:43:36.494804", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HojRJdCTbEdz4VD0jY0AEj2hdryEDUdV7"], "labels": [], "masters": [], "uuids": ["6a7c86ad-e6ee-49ed-b9c6-7d4073796d1a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:36.335252", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154791 s, 745 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154791 s, 745 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.785 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8', 'wwn-0x5000c50087b8cd81'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1', 'wwn-0x5000c50087b8cd81-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8cd81', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2W8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8", "wwn-0x5000c50087b8cd81"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2W8-part1", "wwn-0x5000c50087b8cd81-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2W8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8cd81"}}, "skip_reason": 2020-06-03T18:43:36.787 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-03T18:43:36.911 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi112.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.112969", "end": "2020-06-03 18:43:36.544274", "rc": 0, "start": "2020-06-03 18:43:36.431305", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.948 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:36 +0000 (0:00:00.318) 0:02:15.913 ******** 2020-06-03T18:43:36.968 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:36.969 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ld7T9Z-2Plu-y43P-Xv5R-Sa2Q-FAjr-e0Jc5y', 'nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN', 'nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050336", "end": "2020-06-03 18:43:36.685585", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ld7T9Z-2Plu-y43P-Xv5R-Sa2Q-FAjr-e0Jc5y", "nvme-INTEL_SSDPEDMD400G4_CVFT534300NV400BGN", "nvme-nvme.8086-435646543533343330304e5634303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:36.635249", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0456451 s, 2.5 GB/s", "std 2020-06-03T18:43:36.971 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0456451 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:36.972 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:36.984 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.154 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054269", "end": "2020-06-03 18:43:36.793260", "rc": 0, "start": "2020-06-03 18:43:36.738991", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:37.195 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.247) 0:02:16.161 ******** 2020-06-03T18:43:37.227 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-03T18:43:37.272 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.076) 0:02:16.238 ******** 2020-06-03T18:43:37.284 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:37.286 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HqkMm7qvbNtHzh6y4h9qMvKJoQgHg9v2Z'], 'uuids': ['b1adb7cb-24ac-4d22-995a-3f4c4a1aaa26']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.162835", "end": "2020-06-03 18:43:36.997832", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HqkMm7qvbNtHzh6y4h9qMvKJoQgHg9v2Z"], "labels": [], "masters": [], "uuids": ["b1adb7cb-24ac-4d22-995a-3f4c4a1aaa26"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:36.834997", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158164 s, 729 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158164 s, 729 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:37.288 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.301 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.303 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-06-03T18:43:37.304 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.343 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.071) 0:02:16.309 ******** 2020-06-03T18:43:37.385 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-03T18:43:37.386 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.424 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.080) 0:02:16.390 ******** 2020-06-03T18:43:37.471 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-06-03T18:43:37.473 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.526 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.101) 0:02:16.492 ******** 2020-06-03T18:43:37.568 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.606 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.079) 0:02:16.571 ******** 2020-06-03T18:43:37.619 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:37.621 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HO9Uh7w4n9ijkPelLt92yy2JpOkXeXnqp'], 'uuids': ['c4009b40-c05a-47bd-a8a8-6b5f0b6812bc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159351", "end": "2020-06-03 18:43:37.313532", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HO9Uh7w4n9ijkPelLt92yy2JpOkXeXnqp"], "labels": [], "masters": [], "uuids": ["c4009b40-c05a-47bd-a8a8-6b5f0b6812bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:37.154181", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15455 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15455 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:37.623 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.742 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:37.791 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.185) 0:02:16.756 ******** 2020-06-03T18:43:37.852 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-03T18:43:37.853 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'block_used': 13741036, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354072, 'size_available': 927146278912, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354072, "block_size": 4096, "block_total": 240095108, "block_used": 13741036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146278912, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.891 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.100) 0:02:16.857 ******** 2020-06-03T18:43:37.910 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-03T18:43:37.912 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HZZhytcd4Hmycfxfk1YuwFQCrXz0dpC9l'], 'uuids': ['67df8254-05c7-4b94-9626-517ee202d345']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156048", "end": "2020-06-03 18:43:37.611229", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TQoK1bQ63uZsKnWGrvyxs7fCzRceFV2HZZhytcd4Hmycfxfk1YuwFQCrXz0dpC9l"], "labels": [], "masters": [], "uuids": ["67df8254-05c7-4b94-9626-517ee202d345"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-03 18:43:37.455181", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151333 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151333 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:37.929 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-06-03T18:43:37.930 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:37.969 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:37 +0000 (0:00:00.077) 0:02:16.935 ******** 2020-06-03T18:43:38.007 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.036) 0:02:16.971 ******** 2020-06-03T18:43:38.046 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:38.048 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.084 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.077) 0:02:17.049 ******** 2020-06-03T18:43:38.123 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:38.124 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.160 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.076) 0:02:17.125 ******** 2020-06-03T18:43:38.199 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-06-03T18:43:38.201 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.235 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.075) 0:02:17.200 ******** 2020-06-03T18:43:38.274 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-06-03T18:43:38.276 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.152246", "end": "2020-06-03 18:43:37.998976", "rc": 0, "start": "2020-06-03 18:43:37.846730", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:38.311 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.076) 0:02:17.277 ******** 2020-06-03T18:43:38.321 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-06-03T18:43:38.322 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.358 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.047) 0:02:17.324 ******** 2020-06-03T18:43:38.394 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-06-03T18:43:38.396 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.431 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.072) 0:02:17.396 ******** 2020-06-03T18:43:38.523 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-06-03T18:43:38.525 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.062120", "end": "2020-06-03 18:43:38.256756", "rc": 0, "start": "2020-06-03 18:43:38.194636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:38.577 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.145) 0:02:17.542 ******** 2020-06-03T18:43:38.619 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-03T18:43:38.659 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.082) 0:02:17.625 ******** 2020-06-03T18:43:38.711 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-06-03T18:43:38.713 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.753 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.093) 0:02:17.718 ******** 2020-06-03T18:43:38.796 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-03T18:43:38.798 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.835 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.082) 0:02:17.801 ******** 2020-06-03T18:43:38.865 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-06-03T18:43:38.868 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.906 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.070) 0:02:17.871 ******** 2020-06-03T18:43:38.945 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-06-03T18:43:38.948 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:38.989 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:38 +0000 (0:00:00.083) 0:02:17.954 ******** 2020-06-03T18:43:39.021 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-06-03T18:43:39.022 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-03T18:43:39.063 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.074) 0:02:18.029 ******** 2020-06-03T18:43:39.124 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-06-03T18:43:39.126 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-03T18:43:39.167 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.103) 0:02:18.132 ******** 2020-06-03T18:43:39.219 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-03T18:43:39.221 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.280 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.113) 0:02:18.246 ******** 2020-06-03T18:43:39.322 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-06-03T18:43:39.323 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.358 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.078) 0:02:18.324 ******** 2020-06-03T18:43:39.416 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:39.418 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.451 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.093) 0:02:18.417 ******** 2020-06-03T18:43:39.491 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:39.492 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.540 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.088) 0:02:18.505 ******** 2020-06-03T18:43:39.572 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-06-03T18:43:39.574 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.610 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.069) 0:02:18.575 ******** 2020-06-03T18:43:39.639 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-06-03T18:43:39.646 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.676 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.066) 0:02:18.642 ******** 2020-06-03T18:43:39.686 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:39.688 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:39.708 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:39.752 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:39 +0000 (0:00:00.075) 0:02:18.717 ******** 2020-06-03T18:43:39.900 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:39.902 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.117 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-06-03T18:43:40.119 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-03T18:43:40.154 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:40 +0000 (0:00:00.402) 0:02:19.120 ******** 2020-06-03T18:43:40.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:40.166 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.415 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.488 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.669 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.708 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:40 +0000 (0:00:00.553) 0:02:19.673 ******** 2020-06-03T18:43:40.759 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:40.861 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-03T18:43:40.893 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:40 +0000 (0:00:00.185) 0:02:19.858 ******** 2020-06-03T18:43:41.009 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:41.010 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:41.067 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-03T18:43:41.069 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:41.191 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:41.256 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:41.257 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:41.318 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-03T18:43:41.320 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:41.439 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:41.450 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:41.491 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:41 +0000 (0:00:00.597) 0:02:20.456 ******** 2020-06-03T18:43:41.516 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-03T18:43:41.522 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-03T18:43:41.561 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:41 +0000 (0:00:00.070) 0:02:20.527 ******** 2020-06-03T18:43:41.644 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-06-03T18:43:41.646 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:43:41.689 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:41 +0000 (0:00:00.127) 0:02:20.654 ******** 2020-06-03T18:43:41.729 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-03T18:43:41.731 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-06-03T18:43:41.766 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:41 +0000 (0:00:00.076) 0:02:20.731 ******** 2020-06-03T18:43:41.803 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:41 +0000 (0:00:00.037) 0:02:20.769 ******** 2020-06-03T18:43:41.928 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-03T18:43:41.930 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:42.050 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:42.174 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:42.291 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-03T18:43:42.305 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:42.349 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:42 +0000 (0:00:00.545) 0:02:21.314 ******** 2020-06-03T18:43:42.509 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-06-03T18:43:42.511 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:43:42.554 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:42 +0000 (0:00:00.205) 0:02:21.519 ******** 2020-06-03T18:43:42.603 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-06-03T18:43:42.605 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-06-03T18:43:42.642 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:42 +0000 (0:00:00.087) 0:02:21.607 ******** 2020-06-03T18:43:42.895 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi061.front.sepia.ceph.com] => {"changed": true} 2020-06-03T18:43:42.935 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:42 +0000 (0:00:00.293) 0:02:21.901 ******** 2020-06-03T18:43:43.424 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true} 2020-06-03T18:43:43.463 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.528) 0:02:22.429 ******** 2020-06-03T18:43:43.505 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-06-03T18:43:43.506 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-03T18:43:43.554 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.091) 0:02:22.520 ******** 2020-06-03T18:43:43.689 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-03T18:43:43.734 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.179) 0:02:22.699 ******** 2020-06-03T18:43:43.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-06-03T18:43:43.744 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-06-03T18:43:43.783 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.048) 0:02:22.748 ******** 2020-06-03T18:43:43.896 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-06-03T18:43:43.934 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.151) 0:02:22.900 ******** 2020-06-03T18:43:43.943 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-06-03T18:43:43.945 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:43:43.984 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:43 +0000 (0:00:00.049) 0:02:22.950 ******** 2020-06-03T18:43:44.082 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-06-03T18:43:44.121 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:44 +0000 (0:00:00.136) 0:02:23.086 ******** 2020-06-03T18:43:44.350 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-06-03T18:43:44.352 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-06-03T18:43:44.388 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:44 +0000 (0:00:00.267) 0:02:23.354 ******** 2020-06-03T18:43:44.480 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-06-03T18:43:44.518 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:44 +0000 (0:00:00.129) 0:02:23.483 ******** 2020-06-03T18:43:44.777 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-06-03T18:43:44.813 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:44 +0000 (0:00:00.293) 0:02:23.777 ******** 2020-06-03T18:43:44.892 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-06-03T18:43:44.931 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:44 +0000 (0:00:00.119) 0:02:23.896 ******** 2020-06-03T18:43:45.742 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-06-03T18:43:45.744 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:43:45.782 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:45 +0000 (0:00:00.850) 0:02:24.746 ******** 2020-06-03T18:43:45.821 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:43:45.871 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:45 +0000 (0:00:00.089) 0:02:24.836 ******** 2020-06-03T18:43:45.983 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-06-03T18:43:45.985 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046920", "end": "2020-06-03 18:43:45.719787", "rc": 0, "start": "2020-06-03 18:43:45.672867", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-06-03T18:43:46.021 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.150) 0:02:24.987 ******** 2020-06-03T18:43:46.072 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048189", "end": "2020-06-03 18:43:45.710215", "rc": 0, "start": "2020-06-03 18:43:45.662026", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-06-03T18:43:46.116 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.095) 0:02:25.082 ******** 2020-06-03T18:43:46.127 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-06-03T18:43:46.129 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:46.169 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.052) 0:02:25.134 ******** 2020-06-03T18:43:46.178 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:46.216 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.047) 0:02:25.182 ******** 2020-06-03T18:43:46.225 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-06-03T18:43:46.227 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:46.265 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.048) 0:02:25.230 ******** 2020-06-03T18:43:46.274 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:46.313 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:46 +0000 (0:00:00.048) 0:02:25.279 ******** 2020-06-03T18:43:47.169 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-06-03T18:43:47.170 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:43:47.211 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.892) 0:02:26.172 ******** 2020-06-03T18:43:47.223 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:43:47.263 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.054) 0:02:26.228 ******** 2020-06-03T18:43:47.273 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-06-03T18:43:47.275 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:47.312 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.047) 0:02:26.276 ******** 2020-06-03T18:43:47.319 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:47.363 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.052) 0:02:26.328 ******** 2020-06-03T18:43:47.701 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-06-03T18:43:47.734 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.371) 0:02:26.700 ******** 2020-06-03T18:43:47.779 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-06-03T18:43:47.782 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-06-03T18:43:47.820 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.085) 0:02:26.785 ******** 2020-06-03T18:43:47.859 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.039) 0:02:26.825 ******** 2020-06-03T18:43:47.891 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-06-03T18:43:47.893 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:47.945 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:47 +0000 (0:00:00.085) 0:02:26.911 ******** 2020-06-03T18:43:48.038 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-06-03T18:43:48.040 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:41:10 UTC", "ActiveEnterTimestampMonotonic": "24321573", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice systemd-tmpfiles-setup.service sysinit.target network.target systemd-journald.socket basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:41:03 UTC", "AssertTimestampMonotonic": "17888116", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:41:03 UTC", "ConditionTimestampMonotonic": "17888116", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-06-03T18:43:48.042 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1996", "ExecMainStartTimestamp": "Wed 2020-06-03 18:41:10 UTC", "ExecMainStartTimestampMonotonic": "24321561", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:41:03 UTC", "InactiveExitTimestampMonotonic": "17889026", "InvocationID": "d053e7e888264df1957886d2034bad94", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-06-03T18:43:48.043 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1996", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:41:10 UTC", "S 2020-06-03T18:43:48.045 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24321573", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:41:10 UTC", "WatchdogTimestampMonotonic": "24321572", "WatchdogUSec": "0"}} 2020-06-03T18:43:48.080 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.134) 0:02:27.046 ******** 2020-06-03T18:43:48.112 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:48.114 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-03T18:43:48.148 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.068) 0:02:27.114 ******** 2020-06-03T18:43:48.174 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-06-03T18:43:48.179 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-06-03 18:40:59 UTC", "ActiveEnterTimestampMonotonic": "24821862", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount network.target systemd-tmpfiles-setup.service system.slice sysinit.target systemd-journald.socket basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-06-03 18:40:53 UTC", "AssertTimestampMonotonic": "18360095", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-06-03 18:40:53 UTC", "ConditionTimestampMonotonic": "18360094", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-06-03T18:43:48.181 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1818", "ExecMainStartTimestamp": "Wed 2020-06-03 18:40:59 UTC", "ExecMainStartTimestampMonotonic": "24821848", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-06-03 18:40:53 UTC", "InactiveExitTimestampMonotonic": "18360865", "InvocationID": "3e9aee3d841f4c369cfd23f66a592606", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-06-03T18:43:48.183 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1818", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-06-03 18:40:59 UTC", "S 2020-06-03T18:43:48.184 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24821862", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-06-03 18:40:59 UTC", "WatchdogTimestampMonotonic": "24821860", "WatchdogUSec": "0"}} 2020-06-03T18:43:48.222 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.073) 0:02:27.188 ******** 2020-06-03T18:43:48.236 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:48.284 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.061) 0:02:27.249 ******** 2020-06-03T18:43:48.296 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-03T18:43:48.332 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.048) 0:02:27.298 ******** 2020-06-03T18:43:48.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:48.386 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.049) 0:02:27.347 ******** 2020-06-03T18:43:48.390 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:48.392 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:48.431 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.047) 0:02:27.395 ******** 2020-06-03T18:43:48.461 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-06-03T18:43:48.463 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:48.497 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:48 +0000 (0:00:00.067) 0:02:27.462 ******** 2020-06-03T18:43:49.253 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-03T18:43:49.255 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"cache_update_time": 1591209503, "cache_updated": false, "changed": false} 2020-06-03T18:43:49.293 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.795) 0:02:28.258 ******** 2020-06-03T18:43:49.322 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-06-03T18:43:49.324 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:49.361 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.068) 0:02:28.327 ******** 2020-06-03T18:43:49.393 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-03T18:43:49.395 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"cache_update_time": 1591209462, "cache_updated": false, "changed": false} 2020-06-03T18:43:49.438 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.076) 0:02:28.403 ******** 2020-06-03T18:43:49.467 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-06-03T18:43:49.469 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:49.504 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.065) 0:02:28.468 ******** 2020-06-03T18:43:49.521 INFO:teuthology.task.ansible.out: 2020-06-03T18:43:49.522 INFO:teuthology.task.ansible.out:TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-06-03T18:43:49.524 INFO:teuthology.task.ansible.out: ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-06-03T18:43:49.571 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.067) 0:02:28.536 ******** 2020-06-03T18:43:49.654 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-06-03T18:43:49.698 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.127) 0:02:28.664 ******** 2020-06-03T18:43:49.931 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-06-03T18:43:49.971 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:49 +0000 (0:00:00.272) 0:02:28.936 ******** 2020-06-03T18:43:50.094 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-06-03T18:43:50.131 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:50 +0000 (0:00:00.160) 0:02:29.096 ******** 2020-06-03T18:43:51.005 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-06-03T18:43:51.007 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-03T18:43:51.062 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.931) 0:02:30.028 ******** 2020-06-03T18:43:51.072 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-03T18:43:51.111 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.048) 0:02:30.076 ******** 2020-06-03T18:43:51.592 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi112.front.sepia.ceph.com] => {"changed": false} 2020-06-03T18:43:51.628 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.517) 0:02:30.594 ******** 2020-06-03T18:43:51.650 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false} 2020-06-03T18:43:51.686 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.057) 0:02:30.651 ******** 2020-06-03T18:43:51.694 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-06-03T18:43:51.696 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-06-03T18:43:51.732 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.046) 0:02:30.698 ******** 2020-06-03T18:43:51.742 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-06-03T18:43:51.782 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.049) 0:02:30.747 ******** 2020-06-03T18:43:51.885 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-06-03T18:43:51.886 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1591209444.2637982, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1591209442.7710524, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10034, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1591209442.7710524, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3519411719", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-06-03T18:43:51.925 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.142) 0:02:30.890 ******** 2020-06-03T18:43:51.947 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1591209446.0589564, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1591209444.5662107, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10034, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1591209444.5662107, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3519411719", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-06-03T18:43:51.985 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:51 +0000 (0:00:00.060) 0:02:30.951 ******** 2020-06-03T18:43:52.355 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi112.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-06-03T18:43:52.405 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.419) 0:02:31.371 ******** 2020-06-03T18:43:52.417 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-06-03T18:43:52.468 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.062) 0:02:31.433 ******** 2020-06-03T18:43:52.477 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-06-03T18:43:52.479 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-03T18:43:52.519 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.051) 0:02:31.485 ******** 2020-06-03T18:43:52.529 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-03T18:43:52.567 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.047) 0:02:31.532 ******** 2020-06-03T18:43:52.575 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-06-03T18:43:52.577 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:52.613 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.046) 0:02:31.579 ******** 2020-06-03T18:43:52.622 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:52.660 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.046) 0:02:31.626 ******** 2020-06-03T18:43:52.764 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-03T18:43:52.825 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.165) 0:02:31.791 ******** 2020-06-03T18:43:52.834 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-03T18:43:52.872 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:52 +0000 (0:00:00.046) 0:02:31.837 ******** 2020-06-03T18:43:52.975 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-06-03T18:43:52.977 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-03T18:43:53.014 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.141) 0:02:31.979 ******** 2020-06-03T18:43:53.044 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-06-03T18:43:53.045 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.086 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.071) 0:02:32.051 ******** 2020-06-03T18:43:53.101 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-06-03T18:43:53.104 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-03T18:43:53.155 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.068) 0:02:32.120 ******** 2020-06-03T18:43:53.164 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-06-03T18:43:53.167 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.167 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.168 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.169 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.206 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.051) 0:02:32.172 ******** 2020-06-03T18:43:53.214 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-06-03T18:43:53.216 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.255 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.048) 0:02:32.221 ******** 2020-06-03T18:43:53.267 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-06-03T18:43:53.269 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.269 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.270 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.271 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi112.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.313 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.057) 0:02:32.278 ******** 2020-06-03T18:43:53.324 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-06-03T18:43:53.326 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.337 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.339 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi061.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.375 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.062) 0:02:32.340 ******** 2020-06-03T18:43:53.384 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:53.385 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.425 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.050) 0:02:32.391 ******** 2020-06-03T18:43:53.436 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-06-03T18:43:53.439 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.440 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.440 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.451 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.453 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.489 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.063) 0:02:32.454 ******** 2020-06-03T18:43:53.507 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.537 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.047) 0:02:32.502 ******** 2020-06-03T18:43:53.546 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-03T18:43:53.548 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.588 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.050) 0:02:32.553 ******** 2020-06-03T18:43:53.598 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-06-03T18:43:53.599 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.642 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.054) 0:02:32.608 ******** 2020-06-03T18:43:53.654 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-06-03T18:43:53.655 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.692 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.049) 0:02:32.658 ******** 2020-06-03T18:43:53.701 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-06-03T18:43:53.703 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.740 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.047) 0:02:32.705 ******** 2020-06-03T18:43:53.747 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-06-03T18:43:53.749 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.788 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.048) 0:02:32.754 ******** 2020-06-03T18:43:53.797 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-06-03T18:43:53.799 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.839 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.050) 0:02:32.804 ******** 2020-06-03T18:43:53.847 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-06-03T18:43:53.849 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.886 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.046) 0:02:32.851 ******** 2020-06-03T18:43:53.895 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-03T18:43:53.897 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:53.959 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:53 +0000 (0:00:00.073) 0:02:32.924 ******** 2020-06-03T18:43:53.973 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-06-03T18:43:53.975 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.034 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.074) 0:02:32.999 ******** 2020-06-03T18:43:54.042 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-06-03T18:43:54.044 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.083 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.049) 0:02:33.048 ******** 2020-06-03T18:43:54.091 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.137 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.054) 0:02:33.103 ******** 2020-06-03T18:43:54.146 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-03T18:43:54.148 INFO:teuthology.task.ansible.out:skipping: [smithi112.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.182 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.044) 0:02:33.147 ******** 2020-06-03T18:43:54.190 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-06-03T18:43:54.192 INFO:teuthology.task.ansible.out:skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.235 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.053) 0:02:33.201 ******** 2020-06-03T18:43:54.289 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi061.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-03T18:43:54.328 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:54 +0000 (0:00:00.089) 0:02:33.290 ******** 2020-06-03T18:43:54.535 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi112.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011100", "end": "2020-06-03 18:43:54.004393", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-03 18:43:53.993293", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:54.655 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010141", "end": "2020-06-03 18:43:54.384611", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-03 18:43:54.374470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:54.666 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005650", "end": "2020-06-03 18:43:54.293277", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-03 18:43:54.287627", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:54.786 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006121", "end": "2020-06-03 18:43:54.519367", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-03 18:43:54.513246", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:54.795 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005566", "end": "2020-06-03 18:43:54.428774", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-03 18:43:54.423208", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:54.909 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005677", "end": "2020-06-03 18:43:54.651630", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-03 18:43:54.645953", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:55.056 INFO:teuthology.task.ansible.out:ok: [smithi061.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.020422", "end": "2020-06-03 18:43:54.797043", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-03 18:43:54.776621", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:55.218 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:55 +0000 (0:00:00.892) 0:02:34.183 ******** 2020-06-03T18:43:55.226 INFO:teuthology.task.ansible.out:ok: [smithi112.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.168888", "end": "2020-06-03 18:43:54.732686", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-03 18:43:54.563798", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-03T18:43:55.266 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:55 +0000 (0:00:00.048) 0:02:34.232 ******** 2020-06-03T18:43:55.274 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-06-03T18:43:55.313 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:55 +0000 (0:00:00.047) 0:02:34.279 ******** 2020-06-03T18:43:55.366 INFO:teuthology.task.ansible.out:Wednesday 03 June 2020 18:43:55 +0000 (0:00:00.052) 0:02:34.332 ******** 2020-06-03T18:43:55.458 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-06-03T18:43:55.460 INFO:teuthology.task.ansible.out:changed: [smithi061.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-06-03T18:43:55.529 INFO:teuthology.task.ansible.out:changed: [smithi112.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-06-03T18:43:55.629 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-06-03T18:43:55.631 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-06-03T18:43:55.633 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-06-03T18:43:55.635 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi061.front.sepia.ceph.com : ok=114 changed=33 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi112.front.sepia.ceph.com : ok=114 changed=33 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Wednesday 03 June 2020 18:43:55 +0000 (0:00:00.267) 0:02:34.599 ******** =============================================================================== users : Create all admin users with sudo access. ----------------------- 17.51s users : Update authorized_keys using the keys repo --------------------- 13.64s testnode : Ensure packages are not present. ----------------------------- 9.42s testnode : Upgrade packages --------------------------------------------- 8.91s testnode : Install packages --------------------------------------------- 8.04s testnode : Zap all non-root disks --------------------------------------- 7.12s users : Remove revoked users -------------------------------------------- 3.69s testnode : Update apt cache. -------------------------------------------- 2.55s testnode : Install apt keys --------------------------------------------- 2.21s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 2.04s Gathering Facts --------------------------------------------------------- 1.88s users : Update authorized_keys for each user with literal keys ---------- 1.52s common : Update apt cache ----------------------------------------------- 1.17s Gathering Facts --------------------------------------------------------- 1.08s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.98s testnode : Install packages via pip ------------------------------------- 0.93s testnode : Install ntp package on apt based systems. -------------------- 0.89s testnode : List any leftover Ceph artifacts from previous jobs ---------- 0.89s common : Install K 2020-06-03T18:43:55.637 INFO:teuthology.task.ansible.out:erberos Packages (Debian) ----------------------------- 0.88s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.85s 2020-06-03T18:43:56.011 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi061.front.sepia.ceph.com')] 2020-06-03T18:43:56.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:43:56.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:43:56.091 INFO:teuthology.orchestra.run.smithi112:> true 2020-06-03T18:43:56.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-03T18:43:56.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-03T18:43:56.379 INFO:teuthology.orchestra.run.smithi061:> true 2020-06-03T18:43:56.571 INFO:teuthology.run_tasks:Running task clock... 2020-06-03T18:43:56.594 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-06-03T18:43:56.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-03T18:43:56.595 INFO:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-03T18:43:56.650 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-06-03T18:43:56.651 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Command line: ntpd -gq 2020-06-03T18:43:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: proto: precision = 0.123 usec (-23) 2020-06-03T18:43:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-03T18:43:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: restrict ::: KOD does nothing without LIMITED. 2020-06-03T18:43:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen and drop on 0 v6wildcard [::]:123 2020-06-03T18:43:56.652 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-06-03T18:43:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-03T18:43:56.653 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2020-06-03T18:43:56.654 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen normally on 2 lo 127.0.0.1:123 2020-06-03T18:43:56.654 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2020-06-03T18:43:56.654 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen normally on 4 lo [::1]:123 2020-06-03T18:43:56.654 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2020-06-03T18:43:56.654 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:43:56 ntpd[8741]: Listening on routing socket on fd #22 for interface updates 2020-06-03T18:43:57.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:43:57.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:43:57.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:43:57.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:43:57.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:43:57.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:43:57.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:43:57.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:57 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:43:59.650 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:43:59.650 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:43:59.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:43:59.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:43:59.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:43:59.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:43:59.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:43:59.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:43:59.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:43:59.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:43:59.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:43:59.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:43:59 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:01.649 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:01.650 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:01.650 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:01.650 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:01.651 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:01.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:01.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:01.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:01.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:01.652 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:01.653 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:01 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:03.654 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:03.655 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:03.655 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:03.656 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:03.656 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:03.656 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:03.657 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:03.657 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:03.657 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:03.658 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:03.658 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:03.658 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:03.659 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-06-03T18:44:03.659 INFO:teuthology.orchestra.run.smithi112.stderr: 3 Jun 18:44:03 ntpd[8741]: can't open /var/log/ntpstats/loopstats.20200603: Permission denied 2020-06-03T18:44:03.660 INFO:teuthology.orchestra.run.smithi112.stdout: 3 Jun 18:44:03 ntpd[8741]: ntpd: time set +0.322813 s 2020-06-03T18:44:03.660 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time set +0.322813s 2020-06-03T18:44:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2020-06-03T18:44:03.710 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2020-06-03T18:44:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:03.712 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-03T18:44:03.712 INFO:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-03T18:44:03.771 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-06-03T18:44:03.771 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Command line: ntpd -gq 2020-06-03T18:44:03.775 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: proto: precision = 0.126 usec (-23) 2020-06-03T18:44:03.775 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-03T18:44:03.775 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: restrict ::: KOD does nothing without LIMITED. 2020-06-03T18:44:03.775 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen and drop on 0 v6wildcard [::]:123 2020-06-03T18:44:03.775 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-06-03T18:44:03.776 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen normally on 2 lo 127.0.0.1:123 2020-06-03T18:44:03.776 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen normally on 3 enp3s0f1 172.21.15.61:123 2020-06-03T18:44:03.776 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen normally on 4 lo [::1]:123 2020-06-03T18:44:03.776 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8b9%5]:123 2020-06-03T18:44:03.776 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:03 ntpd[8884]: Listening on routing socket on fd #22 for interface updates 2020-06-03T18:44:03.777 INFO:teuthology.orchestra.run.smithi061.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-03T18:44:03.777 INFO:teuthology.orchestra.run.smithi061.stderr:restrict ::: KOD does nothing without LIMITED. 2020-06-03T18:44:04.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:04 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:04.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:04 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:04.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:04 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:04.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:04 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:05.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:05 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:05.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:05 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:05.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:05 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:05.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:05 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:06.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:06.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:06.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:06.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:06.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:06.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:06.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:06.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:06 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:07.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:07.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:07.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:07.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:07.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:07.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:07.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:07.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:07 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:08.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:08.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:08.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:08.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:08.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:08.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:08.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:08.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:08 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:09.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:09.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:09.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:09.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:09.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:09.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:09.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:09.772 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:09 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:10.769 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:10.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:10.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:10.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:10.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-03T18:44:10.770 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: can't open /var/log/ntpstats/rawstats.20200603: Permission denied 2020-06-03T18:44:10.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-03T18:44:10.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: can't open /var/log/ntpstats/peerstats.20200603: Permission denied 2020-06-03T18:44:10.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-06-03T18:44:10.771 INFO:teuthology.orchestra.run.smithi061.stderr: 3 Jun 18:44:10 ntpd[8884]: can't open /var/log/ntpstats/loopstats.20200603: Permission denied 2020-06-03T18:44:10.772 INFO:teuthology.orchestra.run.smithi061.stdout: 3 Jun 18:44:10 ntpd[8884]: ntpd: time set +0.228038 s 2020-06-03T18:44:10.772 INFO:teuthology.orchestra.run.smithi061.stdout:ntpd: time set +0.228038s 2020-06-03T18:44:10.819 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2020-06-03T18:44:10.820 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2020-06-03T18:44:10.820 INFO:teuthology.orchestra.run.smithi061.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:10.820 INFO:teuthology.orchestra.run.smithi061.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:10.820 INFO:teuthology.orchestra.run.smithi061.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:10.820 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-03T18:44:10.821 INFO:teuthology.run_tasks:Running task install... 2020-06-03T18:44:10.839 DEBUG:teuthology.task.install:project ceph 2020-06-03T18:44:10.840 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35'}} 2020-06-03T18:44:10.840 DEBUG:teuthology.task.install:config {'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35'} 2020-06-03T18:44:10.841 INFO:teuthology.task.install:Using flavor: basic 2020-06-03T18:44:10.850 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-06-03T18:44:10.850 INFO:teuthology.task.install:extra packages: [] 2020-06-03T18:44:10.851 INFO:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2020-06-03T18:44:10.902 INFO:teuthology.orchestra.run.smithi112.stdout:Ubuntu 2020-06-03T18:44:10.907 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:44:10.907 INFO:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2020-06-03T18:44:10.908 INFO:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2020-06-03T18:44:10.963 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2020-06-03T18:44:10.968 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:44:10.969 INFO:teuthology.orchestra.run.smithi061:> sudo apt-key list | grep Ceph 2020-06-03T18:44:11.004 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-06-03T18:44:11.058 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-06-03T18:44:11.059 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2020-06-03T18:44:11.060 INFO:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2020-06-03T18:44:11.063 INFO:teuthology.orchestra.run.smithi061.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-06-03T18:44:11.111 INFO:teuthology.orchestra.run.smithi112.stdout:Ubuntu 2020-06-03T18:44:11.116 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:44:11.116 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-06-03T18:44:11.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=230b0bb1973ab49f8bd55921d5cad7dec0204c35 2020-06-03T18:44:11.142 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-06-03T18:44:11.143 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph.com (release key) 2020-06-03T18:44:11.143 INFO:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2020-06-03T18:44:11.196 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2020-06-03T18:44:11.202 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:44:11.202 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-06-03T18:44:11.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=230b0bb1973ab49f8bd55921d5cad7dec0204c35 2020-06-03T18:44:11.320 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default/ 2020-06-03T18:44:11.321 INFO:teuthology.task.install.deb:Package version is 16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:44:11.368 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default/ 2020-06-03T18:44:11.368 INFO:teuthology.task.install.deb:Package version is 16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:44:11.456 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-06-03T18:44:11.494 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-06-03T18:44:11.505 INFO:teuthology.orchestra.run.smithi112:> sudo apt-get update 2020-06-03T18:44:11.542 INFO:teuthology.orchestra.run.smithi061:> sudo apt-get update 2020-06-03T18:44:11.732 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:44:11.816 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic InRelease 2020-06-03T18:44:11.817 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-03T18:44:11.817 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic InRelease 2020-06-03T18:44:11.825 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:44:11.841 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2020-06-03T18:44:11.842 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2020-06-03T18:44:11.844 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-03T18:44:11.865 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic Release.gpg 2020-06-03T18:44:11.866 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic Release.gpg 2020-06-03T18:44:11.906 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:44:11.924 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-06-03T18:44:11.924 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-06-03T18:44:11.932 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-03T18:44:11.959 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,074 B] 2020-06-03T18:44:11.960 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,074 B] 2020-06-03T18:44:11.988 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:44:12.006 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-03T18:44:12.335 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 26.6 kB in 1s (45.2 kB/s) 2020-06-03T18:44:12.390 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 26.6 kB in 1s (39.1 kB/s) 2020-06-03T18:44:13.557 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:44:13.563 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:44:13.599 INFO:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-2021-g230b0bb-1bionic cephadm=16.0.0-2021-g230b0bb-1bionic ceph-mds=16.0.0-2021-g230b0bb-1bionic ceph-mgr=16.0.0-2021-g230b0bb-1bionic ceph-common=16.0.0-2021-g230b0bb-1bionic ceph-fuse=16.0.0-2021-g230b0bb-1bionic ceph-test=16.0.0-2021-g230b0bb-1bionic radosgw=16.0.0-2021-g230b0bb-1bionic python3-rados=16.0.0-2021-g230b0bb-1bionic python3-rgw=16.0.0-2021-g230b0bb-1bionic python3-cephfs=16.0.0-2021-g230b0bb-1bionic python3-rbd=16.0.0-2021-g230b0bb-1bionic libcephfs2=16.0.0-2021-g230b0bb-1bionic libcephfs-dev=16.0.0-2021-g230b0bb-1bionic librados2=16.0.0-2021-g230b0bb-1bionic librbd1=16.0.0-2021-g230b0bb-1bionic rbd-fuse=16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:44:13.611 INFO:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-2021-g230b0bb-1bionic cephadm=16.0.0-2021-g230b0bb-1bionic ceph-mds=16.0.0-2021-g230b0bb-1bionic ceph-mgr=16.0.0-2021-g230b0bb-1bionic ceph-common=16.0.0-2021-g230b0bb-1bionic ceph-fuse=16.0.0-2021-g230b0bb-1bionic ceph-test=16.0.0-2021-g230b0bb-1bionic radosgw=16.0.0-2021-g230b0bb-1bionic python3-rados=16.0.0-2021-g230b0bb-1bionic python3-rgw=16.0.0-2021-g230b0bb-1bionic python3-cephfs=16.0.0-2021-g230b0bb-1bionic python3-rbd=16.0.0-2021-g230b0bb-1bionic libcephfs2=16.0.0-2021-g230b0bb-1bionic libcephfs-dev=16.0.0-2021-g230b0bb-1bionic librados2=16.0.0-2021-g230b0bb-1bionic librbd1=16.0.0-2021-g230b0bb-1bionic rbd-fuse=16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:44:13.675 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2020-06-03T18:44:13.691 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2020-06-03T18:44:13.919 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2020-06-03T18:44:13.919 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2020-06-03T18:44:13.929 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2020-06-03T18:44:13.930 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2020-06-03T18:44:14.073 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2020-06-03T18:44:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-03T18:44:14.074 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-03T18:44:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-06-03T18:44:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-06-03T18:44:14.075 INFO:teuthology.orchestra.run.smithi112.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-06-03T18:44:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-06-03T18:44:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-06-03T18:44:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-03T18:44:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-03T18:44:14.076 INFO:teuthology.orchestra.run.smithi112.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-03T18:44:14.077 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-03T18:44:14.077 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-03T18:44:14.077 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-03T18:44:14.077 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-03T18:44:14.077 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-06-03T18:44:14.078 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-06-03T18:44:14.079 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-06-03T18:44:14.079 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2020-06-03T18:44:14.079 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-06-03T18:44:14.080 INFO:teuthology.orchestra.run.smithi112.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-06-03T18:44:14.080 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-06-03T18:44:14.080 INFO:teuthology.orchestra.run.smithi112.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-06-03T18:44:14.080 INFO:teuthology.orchestra.run.smithi112.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-06-03T18:44:14.080 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-06-03T18:44:14.081 INFO:teuthology.orchestra.run.smithi112.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-06-03T18:44:14.082 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-06-03T18:44:14.082 INFO:teuthology.orchestra.run.smithi112.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-06-03T18:44:14.082 INFO:teuthology.orchestra.run.smithi112.stdout: python3-watchdog python-werkzeug-doc 2020-06-03T18:44:14.112 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2020-06-03T18:44:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-03T18:44:14.113 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-03T18:44:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-06-03T18:44:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-06-03T18:44:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-06-03T18:44:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-06-03T18:44:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-06-03T18:44:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-03T18:44:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-03T18:44:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-03T18:44:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-03T18:44:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-06-03T18:44:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-06-03T18:44:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-06-03T18:44:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-06-03T18:44:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-06-03T18:44:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-06-03T18:44:14.118 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2020-06-03T18:44:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-06-03T18:44:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-06-03T18:44:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-06-03T18:44:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-06-03T18:44:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-06-03T18:44:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-06-03T18:44:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-06-03T18:44:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-06-03T18:44:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-06-03T18:44:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: python3-watchdog python-werkzeug-doc 2020-06-03T18:44:14.174 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2020-06-03T18:44:14.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-06-03T18:44:14.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-03T18:44:14.174 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-03T18:44:14.175 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-06-03T18:44:14.175 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-06-03T18:44:14.175 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-06-03T18:44:14.175 INFO:teuthology.orchestra.run.smithi112.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-06-03T18:44:14.175 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-03T18:44:14.176 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-03T18:44:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-03T18:44:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-03T18:44:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-03T18:44:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-03T18:44:14.177 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados python3-rbd python3-repoze.lru python3-rgw python3-routes 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa python3-scipy python3-setuptools python3-simplegeneric 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplejson python3-singledispatch python3-sklearn 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn-lib python3-sqlalchemy python3-tempita python3-tk python3-tz 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-uritemplate python3-waitress python3-webencodings python3-webob 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket python3-webtest python3-werkzeug radosgw rbd-fuse socat 2020-06-03T18:44:14.178 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet 2020-06-03T18:44:14.179 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2020-06-03T18:44:14.179 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 librbd1 2020-06-03T18:44:14.222 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2020-06-03T18:44:14.223 INFO:teuthology.orchestra.run.smithi061.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-06-03T18:44:14.223 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-03T18:44:14.223 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-03T18:44:14.224 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-06-03T18:44:14.224 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-06-03T18:44:14.224 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-06-03T18:44:14.225 INFO:teuthology.orchestra.run.smithi061.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-06-03T18:44:14.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-06-03T18:44:14.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-06-03T18:44:14.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-03T18:44:14.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-03T18:44:14.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-03T18:44:14.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-03T18:44:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-03T18:44:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-03T18:44:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-03T18:44:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-03T18:44:14.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-03T18:44:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados python3-rbd python3-repoze.lru python3-rgw python3-routes 2020-06-03T18:44:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa python3-scipy python3-setuptools python3-simplegeneric 2020-06-03T18:44:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplejson python3-singledispatch python3-sklearn 2020-06-03T18:44:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sklearn-lib python3-sqlalchemy python3-tempita python3-tk python3-tz 2020-06-03T18:44:14.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-uritemplate python3-waitress python3-webencodings python3-webob 2020-06-03T18:44:14.229 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket python3-webtest python3-werkzeug radosgw rbd-fuse socat 2020-06-03T18:44:14.229 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet 2020-06-03T18:44:14.229 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2020-06-03T18:44:14.230 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 librbd1 2020-06-03T18:44:14.365 INFO:teuthology.orchestra.run.smithi112.stdout:2 upgraded, 109 newly installed, 0 to remove and 47 not upgraded. 2020-06-03T18:44:14.366 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 162 MB of archives. 2020-06-03T18:44:14.366 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 758 MB of additional disk space will be used. 2020-06-03T18:44:14.366 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-06-03T18:44:14.380 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-2021-g230b0bb-1bionic [1,672 kB] 2020-06-03T18:44:14.417 INFO:teuthology.orchestra.run.smithi061.stdout:2 upgraded, 109 newly installed, 0 to remove and 47 not upgraded. 2020-06-03T18:44:14.417 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 162 MB of archives. 2020-06-03T18:44:14.418 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 758 MB of additional disk space will be used. 2020-06-03T18:44:14.418 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-06-03T18:44:14.440 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-2021-g230b0bb-1bionic [1,672 kB] 2020-06-03T18:44:14.447 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-06-03T18:44:14.496 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-06-03T18:44:14.556 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-2021-g230b0bb-1bionic [3,140 kB] 2020-06-03T18:44:14.700 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-2021-g230b0bb-1bionic [463 kB] 2020-06-03T18:44:14.707 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-06-03T18:44:14.707 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-2021-g230b0bb-1bionic [347 kB] 2020-06-03T18:44:14.708 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-06-03T18:44:14.709 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-06-03T18:44:14.720 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-2021-g230b0bb-1bionic [20.6 kB] 2020-06-03T18:44:14.733 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-2021-g230b0bb-1bionic [132 kB] 2020-06-03T18:44:14.737 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-2021-g230b0bb-1bionic [24.2 kB] 2020-06-03T18:44:14.738 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-2021-g230b0bb-1bionic [308 kB] 2020-06-03T18:44:14.754 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-06-03T18:44:14.759 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-2021-g230b0bb-1bionic [2,885 kB] 2020-06-03T18:44:14.760 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-06-03T18:44:14.775 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-06-03T18:44:14.824 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-06-03T18:44:14.826 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-06-03T18:44:14.868 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-06-03T18:44:14.875 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-06-03T18:44:14.875 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-06-03T18:44:14.878 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-06-03T18:44:14.884 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-06-03T18:44:14.897 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-06-03T18:44:14.906 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-06-03T18:44:14.912 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-06-03T18:44:14.920 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-06-03T18:44:14.928 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-06-03T18:44:14.932 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-06-03T18:44:14.937 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-06-03T18:44:14.938 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-06-03T18:44:14.947 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-06-03T18:44:14.954 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-06-03T18:44:14.967 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-06-03T18:44:14.968 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-06-03T18:44:14.975 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-06-03T18:44:14.979 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-06-03T18:44:14.998 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-06-03T18:44:15.007 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-06-03T18:44:15.008 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-06-03T18:44:15.035 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-06-03T18:44:15.035 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-06-03T18:44:15.038 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-06-03T18:44:15.038 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-06-03T18:44:15.039 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-06-03T18:44:15.041 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-2021-g230b0bb-1bionic [99.1 kB] 2020-06-03T18:44:15.042 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-06-03T18:44:15.044 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-06-03T18:44:15.079 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-06-03T18:44:15.079 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-06-03T18:44:15.082 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-06-03T18:44:15.098 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-06-03T18:44:15.099 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-2021-g230b0bb-1bionic [419 kB] 2020-06-03T18:44:15.099 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-06-03T18:44:15.099 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-06-03T18:44:15.100 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-06-03T18:44:15.115 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-06-03T18:44:15.116 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-2021-g230b0bb-1bionic [19.3 MB] 2020-06-03T18:44:15.116 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-06-03T18:44:15.129 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-06-03T18:44:15.149 INFO:teuthology.orchestra.run.smithi061.stdout:Get:26 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-2021-g230b0bb-1bionic [3,140 kB] 2020-06-03T18:44:15.159 INFO:teuthology.orchestra.run.smithi061.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-06-03T18:44:15.160 INFO:teuthology.orchestra.run.smithi061.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-06-03T18:44:15.181 INFO:teuthology.orchestra.run.smithi061.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-06-03T18:44:15.198 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-06-03T18:44:15.198 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-06-03T18:44:15.199 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-06-03T18:44:15.201 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-06-03T18:44:15.207 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-06-03T18:44:15.208 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-06-03T18:44:15.208 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-06-03T18:44:15.240 INFO:teuthology.orchestra.run.smithi061.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-06-03T18:44:15.240 INFO:teuthology.orchestra.run.smithi061.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-06-03T18:44:15.241 INFO:teuthology.orchestra.run.smithi061.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-06-03T18:44:15.246 INFO:teuthology.orchestra.run.smithi061.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-06-03T18:44:15.253 INFO:teuthology.orchestra.run.smithi061.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-06-03T18:44:15.254 INFO:teuthology.orchestra.run.smithi061.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-06-03T18:44:15.255 INFO:teuthology.orchestra.run.smithi061.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-06-03T18:44:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-06-03T18:44:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-06-03T18:44:15.350 INFO:teuthology.orchestra.run.smithi061.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-06-03T18:44:15.351 INFO:teuthology.orchestra.run.smithi061.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-06-03T18:44:15.563 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-06-03T18:44:15.622 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-06-03T18:44:15.623 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-06-03T18:44:15.629 INFO:teuthology.orchestra.run.smithi061.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-06-03T18:44:15.646 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-06-03T18:44:15.647 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-06-03T18:44:15.647 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-06-03T18:44:15.647 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-06-03T18:44:15.648 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-06-03T18:44:15.649 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-06-03T18:44:15.650 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-06-03T18:44:15.680 INFO:teuthology.orchestra.run.smithi061.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-06-03T18:44:15.682 INFO:teuthology.orchestra.run.smithi061.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-06-03T18:44:15.715 INFO:teuthology.orchestra.run.smithi061.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-06-03T18:44:15.715 INFO:teuthology.orchestra.run.smithi061.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-06-03T18:44:15.716 INFO:teuthology.orchestra.run.smithi061.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-06-03T18:44:15.717 INFO:teuthology.orchestra.run.smithi061.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-06-03T18:44:15.717 INFO:teuthology.orchestra.run.smithi061.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-06-03T18:44:15.718 INFO:teuthology.orchestra.run.smithi061.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-06-03T18:44:15.718 INFO:teuthology.orchestra.run.smithi061.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-06-03T18:44:15.726 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-06-03T18:44:15.729 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-06-03T18:44:15.792 INFO:teuthology.orchestra.run.smithi061.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-06-03T18:44:15.794 INFO:teuthology.orchestra.run.smithi061.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-06-03T18:44:15.807 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-06-03T18:44:15.808 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-06-03T18:44:15.814 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-06-03T18:44:15.814 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-06-03T18:44:15.815 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-06-03T18:44:15.816 INFO:teuthology.orchestra.run.smithi061.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-2021-g230b0bb-1bionic [463 kB] 2020-06-03T18:44:15.819 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-06-03T18:44:15.820 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-06-03T18:44:15.821 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-06-03T18:44:15.872 INFO:teuthology.orchestra.run.smithi061.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-06-03T18:44:15.874 INFO:teuthology.orchestra.run.smithi061.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-06-03T18:44:15.875 INFO:teuthology.orchestra.run.smithi061.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-2021-g230b0bb-1bionic [347 kB] 2020-06-03T18:44:15.877 INFO:teuthology.orchestra.run.smithi061.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-06-03T18:44:15.879 INFO:teuthology.orchestra.run.smithi061.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-06-03T18:44:15.879 INFO:teuthology.orchestra.run.smithi061.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-06-03T18:44:15.881 INFO:teuthology.orchestra.run.smithi061.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-06-03T18:44:15.882 INFO:teuthology.orchestra.run.smithi061.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-06-03T18:44:15.883 INFO:teuthology.orchestra.run.smithi061.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-2021-g230b0bb-1bionic [20.6 kB] 2020-06-03T18:44:15.883 INFO:teuthology.orchestra.run.smithi061.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-2021-g230b0bb-1bionic [132 kB] 2020-06-03T18:44:15.884 INFO:teuthology.orchestra.run.smithi061.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-06-03T18:44:15.885 INFO:teuthology.orchestra.run.smithi061.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-2021-g230b0bb-1bionic [24.2 kB] 2020-06-03T18:44:15.887 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-06-03T18:44:15.889 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-06-03T18:44:15.891 INFO:teuthology.orchestra.run.smithi061.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-2021-g230b0bb-1bionic [308 kB] 2020-06-03T18:44:15.916 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-2021-g230b0bb-1bionic [5,606 kB] 2020-06-03T18:44:15.929 INFO:teuthology.orchestra.run.smithi061.stdout:Get:65 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-2021-g230b0bb-1bionic [2,885 kB] 2020-06-03T18:44:15.953 INFO:teuthology.orchestra.run.smithi061.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-06-03T18:44:15.957 INFO:teuthology.orchestra.run.smithi061.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-06-03T18:44:15.970 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-06-03T18:44:15.970 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-06-03T18:44:15.971 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-06-03T18:44:15.971 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-06-03T18:44:15.973 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-06-03T18:44:15.975 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-06-03T18:44:15.976 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-06-03T18:44:15.976 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-06-03T18:44:16.034 INFO:teuthology.orchestra.run.smithi061.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-06-03T18:44:16.034 INFO:teuthology.orchestra.run.smithi061.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-06-03T18:44:16.035 INFO:teuthology.orchestra.run.smithi061.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-06-03T18:44:16.036 INFO:teuthology.orchestra.run.smithi061.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-06-03T18:44:16.039 INFO:teuthology.orchestra.run.smithi061.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-06-03T18:44:16.042 INFO:teuthology.orchestra.run.smithi061.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-06-03T18:44:16.043 INFO:teuthology.orchestra.run.smithi061.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-06-03T18:44:16.044 INFO:teuthology.orchestra.run.smithi061.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-06-03T18:44:16.049 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-06-03T18:44:16.051 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-06-03T18:44:16.115 INFO:teuthology.orchestra.run.smithi061.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-06-03T18:44:16.117 INFO:teuthology.orchestra.run.smithi061.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-06-03T18:44:16.121 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-2021-g230b0bb-1bionic [141 kB] 2020-06-03T18:44:16.126 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-2021-g230b0bb-1bionic [1,351 kB] 2020-06-03T18:44:16.130 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-06-03T18:44:16.131 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-06-03T18:44:16.132 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-06-03T18:44:16.185 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-2021-g230b0bb-1bionic [6,574 kB] 2020-06-03T18:44:16.196 INFO:teuthology.orchestra.run.smithi061.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-06-03T18:44:16.197 INFO:teuthology.orchestra.run.smithi061.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-06-03T18:44:16.198 INFO:teuthology.orchestra.run.smithi061.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-06-03T18:44:16.231 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-06-03T18:44:16.232 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-06-03T18:44:16.233 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-06-03T18:44:16.235 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-06-03T18:44:16.236 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-06-03T18:44:16.237 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-06-03T18:44:16.237 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-06-03T18:44:16.292 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-06-03T18:44:16.293 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-06-03T18:44:16.298 INFO:teuthology.orchestra.run.smithi061.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-06-03T18:44:16.372 INFO:teuthology.orchestra.run.smithi061.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-06-03T18:44:16.373 INFO:teuthology.orchestra.run.smithi061.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-06-03T18:44:16.381 INFO:teuthology.orchestra.run.smithi061.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-06-03T18:44:16.381 INFO:teuthology.orchestra.run.smithi061.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-06-03T18:44:16.382 INFO:teuthology.orchestra.run.smithi061.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-06-03T18:44:16.382 INFO:teuthology.orchestra.run.smithi061.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-06-03T18:44:16.384 INFO:teuthology.orchestra.run.smithi061.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-06-03T18:44:16.384 INFO:teuthology.orchestra.run.smithi061.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-06-03T18:44:16.434 INFO:teuthology.orchestra.run.smithi061.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-2021-g230b0bb-1bionic [99.1 kB] 2020-06-03T18:44:16.471 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-2021-g230b0bb-1bionic [24.6 MB] 2020-06-03T18:44:16.483 INFO:teuthology.orchestra.run.smithi061.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-2021-g230b0bb-1bionic [419 kB] 2020-06-03T18:44:16.498 INFO:teuthology.orchestra.run.smithi061.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-2021-g230b0bb-1bionic [19.3 MB] 2020-06-03T18:44:17.535 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-2021-g230b0bb-1bionic [3,804 B] 2020-06-03T18:44:17.536 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-2021-g230b0bb-1bionic [543 kB] 2020-06-03T18:44:17.545 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-2021-g230b0bb-1bionic [1,899 kB] 2020-06-03T18:44:17.608 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-2021-g230b0bb-1bionic [41.3 kB] 2020-06-03T18:44:17.613 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-2021-g230b0bb-1bionic [39.5 kB] 2020-06-03T18:44:17.613 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-2021-g230b0bb-1bionic [3,120 kB] 2020-06-03T18:44:17.713 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-2021-g230b0bb-1bionic [34.1 kB] 2020-06-03T18:44:17.714 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-2021-g230b0bb-1bionic [7,886 kB] 2020-06-03T18:44:17.974 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-2021-g230b0bb-1bionic [3,924 B] 2020-06-03T18:44:17.975 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-2021-g230b0bb-1bionic [22.3 kB] 2020-06-03T18:44:18.094 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-2021-g230b0bb-1bionic [46.8 MB] 2020-06-03T18:44:20.361 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-2021-g230b0bb-1bionic [17.9 kB] 2020-06-03T18:44:20.361 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-2021-g230b0bb-1bionic [8,788 kB] 2020-06-03T18:44:20.763 INFO:teuthology.orchestra.run.smithi112.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-2021-g230b0bb-1bionic [79.0 kB] 2020-06-03T18:44:20.919 INFO:teuthology.orchestra.run.smithi061.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-2021-g230b0bb-1bionic [5,606 kB] 2020-06-03T18:44:21.121 INFO:teuthology.orchestra.run.smithi061.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-2021-g230b0bb-1bionic [141 kB] 2020-06-03T18:44:21.134 INFO:teuthology.orchestra.run.smithi061.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-2021-g230b0bb-1bionic [1,351 kB] 2020-06-03T18:44:21.177 INFO:teuthology.orchestra.run.smithi061.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-2021-g230b0bb-1bionic [6,574 kB] 2020-06-03T18:44:21.413 INFO:teuthology.orchestra.run.smithi061.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-2021-g230b0bb-1bionic [24.6 MB] 2020-06-03T18:44:21.464 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 162 MB in 7s (24.7 MB/s) 2020-06-03T18:44:21.591 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-uritemplate. 2020-06-03T18:44:21.641 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137516 files and directories currently installed.) 2020-06-03T18:44:21.645 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-06-03T18:44:21.689 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-06-03T18:44:21.967 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-06-03T18:44:21.982 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-06-03T18:44:22.067 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-03T18:44:22.285 INFO:teuthology.orchestra.run.smithi061.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-2021-g230b0bb-1bionic [3,804 B] 2020-06-03T18:44:22.285 INFO:teuthology.orchestra.run.smithi061.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-2021-g230b0bb-1bionic [543 kB] 2020-06-03T18:44:22.299 INFO:teuthology.orchestra.run.smithi061.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-2021-g230b0bb-1bionic [1,899 kB] 2020-06-03T18:44:22.366 INFO:teuthology.orchestra.run.smithi061.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-2021-g230b0bb-1bionic [41.3 kB] 2020-06-03T18:44:22.366 INFO:teuthology.orchestra.run.smithi061.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-2021-g230b0bb-1bionic [39.5 kB] 2020-06-03T18:44:22.367 INFO:teuthology.orchestra.run.smithi061.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-2021-g230b0bb-1bionic [3,120 kB] 2020-06-03T18:44:22.401 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-librbd1_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:22.473 INFO:teuthology.orchestra.run.smithi061.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-2021-g230b0bb-1bionic [34.1 kB] 2020-06-03T18:44:22.475 INFO:teuthology.orchestra.run.smithi061.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-2021-g230b0bb-1bionic [7,886 kB] 2020-06-03T18:44:22.537 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (16.0.0-2021-g230b0bb-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-03T18:44:22.747 INFO:teuthology.orchestra.run.smithi061.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-2021-g230b0bb-1bionic [3,924 B] 2020-06-03T18:44:22.747 INFO:teuthology.orchestra.run.smithi061.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-2021-g230b0bb-1bionic [22.3 kB] 2020-06-03T18:44:22.748 INFO:teuthology.orchestra.run.smithi061.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-2021-g230b0bb-1bionic [46.8 MB] 2020-06-03T18:44:23.089 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-librados2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:23.211 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (16.0.0-2021-g230b0bb-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-03T18:44:23.840 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2020-06-03T18:44:23.855 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:23.898 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:24.200 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2020-06-03T18:44:24.215 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-python3-rados_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:24.250 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:24.349 INFO:teuthology.orchestra.run.smithi061.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-2021-g230b0bb-1bionic [17.9 kB] 2020-06-03T18:44:24.350 INFO:teuthology.orchestra.run.smithi061.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-2021-g230b0bb-1bionic [8,788 kB] 2020-06-03T18:44:24.570 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-06-03T18:44:24.584 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:24.628 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:24.646 INFO:teuthology.orchestra.run.smithi061.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/fix_issue_43536/230b0bb1973ab49f8bd55921d5cad7dec0204c35/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-2021-g230b0bb-1bionic [79.0 kB] 2020-06-03T18:44:24.931 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2020-06-03T18:44:24.944 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:24.989 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:25.325 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2020-06-03T18:44:25.342 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:25.362 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 162 MB in 10s (15.6 MB/s) 2020-06-03T18:44:25.384 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:25.506 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-uritemplate. 2020-06-03T18:44:25.555 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137516 files and directories currently installed.) 2020-06-03T18:44:25.559 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-06-03T18:44:25.596 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-06-03T18:44:25.722 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2020-06-03T18:44:25.736 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-06-03T18:44:25.837 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-06-03T18:44:26.057 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-06-03T18:44:26.074 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-06-03T18:44:26.081 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2020-06-03T18:44:26.098 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:26.139 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:26.157 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-03T18:44:26.458 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../002-librbd1_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:26.525 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-06-03T18:44:26.538 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-06-03T18:44:26.575 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-03T18:44:26.578 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librbd1 (16.0.0-2021-g230b0bb-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-03T18:44:26.870 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-06-03T18:44:26.884 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-06-03T18:44:26.929 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-03T18:44:27.121 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../003-librados2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:27.249 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librados2 (16.0.0-2021-g230b0bb-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-03T18:44:27.281 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2020-06-03T18:44:27.296 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-librgw2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:27.339 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:27.869 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs2. 2020-06-03T18:44:27.883 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2020-06-03T18:44:27.884 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:27.897 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:27.919 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:27.934 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:28.221 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rados. 2020-06-03T18:44:28.236 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../005-python3-rados_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:28.261 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-06-03T18:44:28.278 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-06-03T18:44:28.288 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rados (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:28.320 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-03T18:44:28.564 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-06-03T18:44:28.574 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-06-03T18:44:28.614 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-06-03T18:44:28.641 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-06-03T18:44:28.654 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:28.691 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-argparse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:28.993 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cephfs. 2020-06-03T18:44:29.001 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0. 2020-06-03T18:44:29.008 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:29.016 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-06-03T18:44:29.052 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cephfs (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:29.059 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-06-03T18:44:29.313 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-common. 2020-06-03T18:44:29.326 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:29.362 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:29.370 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2020-06-03T18:44:29.384 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:29.428 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:29.732 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-prettytable. 2020-06-03T18:44:29.746 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-06-03T18:44:29.791 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-06-03T18:44:29.832 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2020-06-03T18:44:29.846 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-ceph-common_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:29.890 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:30.034 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rbd. 2020-06-03T18:44:30.049 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:30.093 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rbd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:30.465 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-06-03T18:44:30.477 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-06-03T18:44:30.512 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-03T18:44:30.815 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-06-03T18:44:30.831 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-06-03T18:44:30.874 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-03T18:44:31.226 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librgw2. 2020-06-03T18:44:31.241 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../013-librgw2_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:31.284 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librgw2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:31.729 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2020-06-03T18:44:31.743 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-ceph-base_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:31.796 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:31.811 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rgw. 2020-06-03T18:44:31.825 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:31.870 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:32.164 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-06-03T18:44:32.179 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-06-03T18:44:32.214 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-03T18:44:32.508 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-06-03T18:44:32.530 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-06-03T18:44:32.568 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-06-03T18:44:32.859 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-dateutil. 2020-06-03T18:44:32.874 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-06-03T18:44:32.917 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-06-03T18:44:32.928 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liboath0. 2020-06-03T18:44:32.942 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-06-03T18:44:32.986 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-06-03T18:44:33.281 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libradosstriper1. 2020-06-03T18:44:33.294 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:33.295 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-asn1crypto. 2020-06-03T18:44:33.310 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-06-03T18:44:33.322 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libradosstriper1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:33.346 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-06-03T18:44:33.618 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cffi-backend. 2020-06-03T18:44:33.633 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-06-03T18:44:33.676 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-06-03T18:44:33.775 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-common. 2020-06-03T18:44:33.792 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../019-ceph-common_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:33.834 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:34.000 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cryptography. 2020-06-03T18:44:34.009 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-06-03T18:44:34.045 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-03T18:44:34.408 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2020-06-03T18:44:34.424 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-06-03T18:44:34.466 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-06-03T18:44:34.726 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-06-03T18:44:34.740 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:34.776 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:35.506 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2020-06-03T18:44:35.522 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-06-03T18:44:35.655 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-base. 2020-06-03T18:44:35.664 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-06-03T18:44:35.671 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../020-ceph-base_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:35.732 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-base (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:35.950 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2020-06-03T18:44:35.964 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-06-03T18:44:36.009 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-06-03T18:44:36.345 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jwt. 2020-06-03T18:44:37.104 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-06-03T18:44:37.105 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-06-03T18:44:37.105 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2020-06-03T18:44:37.105 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-06-03T18:44:37.105 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-06-03T18:44:37.126 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2020-06-03T18:44:37.140 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-06-03T18:44:37.168 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-dateutil. 2020-06-03T18:44:37.184 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-06-03T18:44:37.184 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-06-03T18:44:37.227 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-06-03T18:44:37.495 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2020-06-03T18:44:37.508 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-06-03T18:44:37.537 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-06-03T18:44:37.604 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-asn1crypto. 2020-06-03T18:44:37.618 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-06-03T18:44:37.654 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-06-03T18:44:37.798 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2020-06-03T18:44:37.814 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-06-03T18:44:37.857 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-06-03T18:44:37.910 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cffi-backend. 2020-06-03T18:44:37.929 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-06-03T18:44:37.960 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-06-03T18:44:38.184 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2020-06-03T18:44:38.198 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-06-03T18:44:38.235 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-06-03T18:44:38.271 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cryptography. 2020-06-03T18:44:38.285 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-06-03T18:44:38.330 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-03T18:44:38.529 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2020-06-03T18:44:38.543 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-06-03T18:44:38.579 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-06-03T18:44:38.699 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-openssl. 2020-06-03T18:44:38.713 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-06-03T18:44:38.758 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-06-03T18:44:38.891 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bs4. 2020-06-03T18:44:38.905 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-06-03T18:44:38.948 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-06-03T18:44:39.035 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-06-03T18:44:39.049 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:39.093 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-modules-core (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:39.252 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2020-06-03T18:44:39.266 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-06-03T18:44:39.323 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-06-03T18:44:39.414 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bcrypt. 2020-06-03T18:44:39.431 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-06-03T18:44:39.473 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-06-03T18:44:39.605 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2020-06-03T18:44:39.618 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-06-03T18:44:39.655 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-06-03T18:44:39.776 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cherrypy3. 2020-06-03T18:44:39.790 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-06-03T18:44:39.834 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-06-03T18:44:39.949 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2020-06-03T18:44:39.963 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-06-03T18:44:40.007 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-03T18:44:40.145 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jwt. 2020-06-03T18:44:40.162 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-06-03T18:44:40.204 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-06-03T18:44:40.343 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-06-03T18:44:40.357 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-06-03T18:44:40.394 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-06-03T18:44:40.523 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-logutils. 2020-06-03T18:44:40.538 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-06-03T18:44:40.582 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-06-03T18:44:40.713 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2020-06-03T18:44:40.726 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-06-03T18:44:40.763 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-06-03T18:44:40.918 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-markupsafe. 2020-06-03T18:44:40.932 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-06-03T18:44:40.976 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-06-03T18:44:41.058 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2020-06-03T18:44:41.074 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-06-03T18:44:41.116 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-06-03T18:44:41.279 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-mako. 2020-06-03T18:44:41.293 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-06-03T18:44:41.337 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-06-03T18:44:41.452 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2020-06-03T18:44:41.467 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-06-03T18:44:41.510 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-06-03T18:44:41.656 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplegeneric. 2020-06-03T18:44:41.670 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-06-03T18:44:41.731 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-06-03T18:44:41.890 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2020-06-03T18:44:41.904 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-06-03T18:44:41.948 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-03T18:44:42.034 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-singledispatch. 2020-06-03T18:44:42.049 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-06-03T18:44:42.092 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-06-03T18:44:42.310 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2020-06-03T18:44:42.324 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:42.368 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:42.429 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webob. 2020-06-03T18:44:42.442 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-06-03T18:44:42.487 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-06-03T18:44:42.823 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bs4. 2020-06-03T18:44:42.837 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-06-03T18:44:42.871 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2020-06-03T18:44:42.881 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-06-03T18:44:42.884 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:42.929 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:43.192 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-waitress. 2020-06-03T18:44:43.206 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-06-03T18:44:43.251 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-06-03T18:44:43.570 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tempita. 2020-06-03T18:44:43.587 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-06-03T18:44:43.629 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-06-03T18:44:43.850 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2020-06-03T18:44:43.864 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:43.908 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:43.931 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-paste. 2020-06-03T18:44:43.945 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-06-03T18:44:43.990 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-03T18:44:44.318 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-06-03T18:44:44.334 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-06-03T18:44:44.376 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-06-03T18:44:44.712 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastedeploy. 2020-06-03T18:44:44.726 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-06-03T18:44:44.770 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-06-03T18:44:45.081 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webtest. 2020-06-03T18:44:45.095 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-06-03T18:44:45.131 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-06-03T18:44:45.451 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pecan. 2020-06-03T18:44:45.468 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-06-03T18:44:45.509 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-06-03T18:44:45.864 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-werkzeug. 2020-06-03T18:44:45.878 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-06-03T18:44:45.914 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-03T18:44:46.259 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr. 2020-06-03T18:44:46.275 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:46.317 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:46.626 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2020-06-03T18:44:46.636 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-ceph_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:46.681 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:46.778 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mon. 2020-06-03T18:44:46.794 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:47.000 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2020-06-03T18:44:47.013 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:47.067 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:47.456 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2020-06-03T18:44:47.468 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:47.495 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mon (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:47.522 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:48.089 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2020-06-03T18:44:48.104 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-cephadm_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:48.147 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:48.482 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-osd. 2020-06-03T18:44:48.483 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-06-03T18:44:48.495 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:48.496 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:48.534 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:48.540 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-osd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:48.853 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2020-06-03T18:44:48.867 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-1_all.deb ... 2020-06-03T18:44:48.911 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-06-03T18:44:49.206 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2020-06-03T18:44:49.222 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-python3-routes_2.4.1-1_all.deb ... 2020-06-03T18:44:49.264 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-06-03T18:44:49.605 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-06-03T18:44:49.621 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-ceph-mgr-dashboard_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:49.663 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:50.531 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-06-03T18:44:50.544 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-ceph-mgr-diskprediction-cloud_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:50.572 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph. 2020-06-03T18:44:50.587 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../048-ceph_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:50.588 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:50.629 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:50.925 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-numpy. 2020-06-03T18:44:50.939 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-06-03T18:44:50.984 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-03T18:44:50.985 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-fuse. 2020-06-03T18:44:50.996 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:51.040 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:51.430 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mds. 2020-06-03T18:44:51.444 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:51.487 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mds (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:51.581 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-decorator. 2020-06-03T18:44:51.595 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-06-03T18:44:51.639 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-06-03T18:44:51.975 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-scipy. 2020-06-03T18:44:51.990 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-06-03T18:44:52.029 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephadm. 2020-06-03T18:44:52.042 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-06-03T18:44:52.044 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../051-cephadm_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:44:52.088 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:52.407 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-06-03T18:44:52.421 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:52.473 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:52.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-repoze.lru. 2020-06-03T18:44:52.801 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-1_all.deb ... 2020-06-03T18:44:52.843 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-06-03T18:44:53.137 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-routes. 2020-06-03T18:44:53.153 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../054-python3-routes_2.4.1-1_all.deb ... 2020-06-03T18:44:53.159 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-06-03T18:44:53.176 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-06-03T18:44:53.196 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-06-03T18:44:53.217 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-06-03T18:44:53.518 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-06-03T18:44:53.533 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../055-ceph-mgr-dashboard_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:53.575 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-dashboard (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:53.721 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2020-06-03T18:44:53.735 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-06-03T18:44:53.779 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.11-1) ... 2020-06-03T18:44:54.149 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2020-06-03T18:44:54.163 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-06-03T18:44:54.207 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-06-03T18:44:54.492 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-06-03T18:44:54.508 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../056-ceph-mgr-diskprediction-cloud_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:54.550 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:54.789 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-06-03T18:44:54.802 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:54.839 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:54.886 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-numpy. 2020-06-03T18:44:54.903 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-06-03T18:44:54.945 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-03T18:44:55.559 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-decorator. 2020-06-03T18:44:55.573 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-06-03T18:44:55.617 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-06-03T18:44:55.962 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-scipy. 2020-06-03T18:44:55.976 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-06-03T18:44:56.017 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-monotonic. 2020-06-03T18:44:56.020 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-06-03T18:44:56.033 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-06-03T18:44:56.081 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-06-03T18:44:56.412 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-fasteners. 2020-06-03T18:44:56.426 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-06-03T18:44:56.471 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-06-03T18:44:56.756 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-httplib2. 2020-06-03T18:44:56.773 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-06-03T18:44:56.815 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-03T18:44:57.101 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2020-06-03T18:44:57.117 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-06-03T18:44:57.159 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-06-03T18:44:57.171 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-06-03T18:44:57.185 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-06-03T18:44:57.238 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-06-03T18:44:57.529 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-06-03T18:44:57.546 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-06-03T18:44:57.587 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-03T18:44:57.699 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-joblib. 2020-06-03T18:44:57.714 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-06-03T18:44:57.757 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-joblib (0.11-1) ... 2020-06-03T18:44:57.907 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2020-06-03T18:44:57.921 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-06-03T18:44:57.965 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-06-03T18:44:58.118 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn. 2020-06-03T18:44:58.125 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-06-03T18:44:58.169 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-06-03T18:44:58.285 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-06-03T18:44:58.302 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-06-03T18:44:58.343 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-03T18:44:58.692 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-06-03T18:44:58.708 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:44:58.750 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:44:58.856 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-oauth2client. 2020-06-03T18:44:58.869 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-06-03T18:44:58.914 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-06-03T18:44:59.251 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-setuptools. 2020-06-03T18:44:59.266 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-06-03T18:44:59.309 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-06-03T18:44:59.729 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2020-06-03T18:44:59.744 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-06-03T18:44:59.809 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-06-03T18:44:59.862 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-monotonic. 2020-06-03T18:44:59.877 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-06-03T18:44:59.928 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-06-03T18:45:00.116 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2020-06-03T18:45:00.131 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-06-03T18:45:00.174 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-03T18:45:00.240 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-fasteners. 2020-06-03T18:45:00.255 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-06-03T18:45:00.298 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-06-03T18:45:00.586 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-06-03T18:45:00.600 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:45:00.601 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-httplib2. 2020-06-03T18:45:00.618 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-06-03T18:45:00.645 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:00.659 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-03T18:45:00.945 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1. 2020-06-03T18:45:00.962 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-06-03T18:45:00.964 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-json-pointer. 2020-06-03T18:45:00.980 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-06-03T18:45:01.003 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-06-03T18:45:01.030 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-06-03T18:45:01.350 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-06-03T18:45:01.365 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-06-03T18:45:01.417 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jsonpatch. 2020-06-03T18:45:01.433 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-06-03T18:45:01.434 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-03T18:45:01.478 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-03T18:45:01.745 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rsa. 2020-06-03T18:45:01.762 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-06-03T18:45:01.795 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-06-03T18:45:01.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-06-03T18:45:01.810 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:45:01.853 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-rook (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:02.124 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-06-03T18:45:02.138 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-06-03T18:45:02.182 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-03T18:45:02.190 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig4:amd64. 2020-06-03T18:45:02.204 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-06-03T18:45:02.248 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-06-03T18:45:02.643 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2020-06-03T18:45:02.645 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-oauth2client. 2020-06-03T18:45:02.661 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-06-03T18:45:02.661 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-06-03T18:45:02.702 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-06-03T18:45:02.779 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-06-03T18:45:03.039 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-setuptools. 2020-06-03T18:45:03.055 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-06-03T18:45:03.056 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2020-06-03T18:45:03.070 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-06-03T18:45:03.097 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-06-03T18:45:03.106 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-06-03T18:45:03.376 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2020-06-03T18:45:03.389 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-06-03T18:45:03.434 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-06-03T18:45:03.518 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-websocket. 2020-06-03T18:45:03.531 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-06-03T18:45:03.576 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-06-03T18:45:03.821 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-06-03T18:45:03.837 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-06-03T18:45:03.879 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-03T18:45:03.895 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-kubernetes. 2020-06-03T18:45:03.910 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-06-03T18:45:03.953 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-03T18:45:04.157 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2020-06-03T18:45:04.173 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-06-03T18:45:04.216 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-06-03T18:45:04.383 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-06-03T18:45:04.397 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:45:04.441 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:04.562 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2020-06-03T18:45:04.576 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-ceph-test_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:04.620 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:04.760 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-json-pointer. 2020-06-03T18:45:04.776 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-06-03T18:45:04.818 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-06-03T18:45:05.205 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jsonpatch. 2020-06-03T18:45:05.220 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-06-03T18:45:05.266 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-03T18:45:05.516 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-06-03T18:45:05.532 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-2021-g230b0bb-1bionic_all.deb ... 2020-06-03T18:45:05.566 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-rook (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:05.836 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libonig4:amd64. 2020-06-03T18:45:05.852 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-06-03T18:45:05.886 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-06-03T18:45:06.247 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjq1:amd64. 2020-06-03T18:45:06.261 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-06-03T18:45:06.383 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-06-03T18:45:06.610 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package jq. 2020-06-03T18:45:06.624 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-06-03T18:45:06.660 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-06-03T18:45:06.896 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package socat. 2020-06-03T18:45:06.913 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-06-03T18:45:06.954 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-06-03T18:45:07.692 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-06-03T18:45:07.708 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-06-03T18:45:07.750 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-03T18:45:08.028 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package xmlstarlet. 2020-06-03T18:45:08.047 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-06-03T18:45:08.082 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2020-06-03T18:45:08.086 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-06-03T18:45:08.097 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:08.140 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:08.432 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-test. 2020-06-03T18:45:08.447 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../085-ceph-test_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:08.460 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-attr. 2020-06-03T18:45:08.474 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-python-attr_17.4.0-2_all.deb ... 2020-06-03T18:45:08.490 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-test (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:08.518 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-attr (17.4.0-2) ... 2020-06-03T18:45:08.846 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-funcsigs. 2020-06-03T18:45:08.860 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-python-funcsigs_1.0.2-4_all.deb ... 2020-06-03T18:45:08.904 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-06-03T18:45:09.207 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pluggy. 2020-06-03T18:45:09.270 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-python-pluggy_0.6.0-1_all.deb ... 2020-06-03T18:45:09.271 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-06-03T18:45:09.585 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-py. 2020-06-03T18:45:09.601 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-python-py_1.5.2-1_all.deb ... 2020-06-03T18:45:09.651 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-py (1.5.2-1) ... 2020-06-03T18:45:09.996 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pytest. 2020-06-03T18:45:10.013 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-python-pytest_3.3.2-2_all.deb ... 2020-06-03T18:45:10.054 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-06-03T18:45:10.541 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-attr. 2020-06-03T18:45:10.558 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-python3-attr_17.4.0-2_all.deb ... 2020-06-03T18:45:10.633 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-06-03T18:45:10.944 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cycler. 2020-06-03T18:45:10.959 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-python3-cycler_0.10.0-1_all.deb ... 2020-06-03T18:45:11.002 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-06-03T18:45:11.314 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webencodings. 2020-06-03T18:45:11.328 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-python3-webencodings_0.5-2_all.deb ... 2020-06-03T18:45:11.364 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-06-03T18:45:11.608 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-html5lib. 2020-06-03T18:45:11.621 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-python3-html5lib_0.999999999-1_all.deb ... 2020-06-03T18:45:11.658 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-06-03T18:45:11.952 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs-dev. 2020-06-03T18:45:11.972 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:12.002 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs-dev (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:12.003 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-06-03T18:45:12.024 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-06-03T18:45:12.061 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-03T18:45:12.313 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-attr. 2020-06-03T18:45:12.328 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../087-python-attr_17.4.0-2_all.deb ... 2020-06-03T18:45:12.371 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-attr (17.4.0-2) ... 2020-06-03T18:45:12.456 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyparsing. 2020-06-03T18:45:12.473 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-06-03T18:45:12.514 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-03T18:45:12.708 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-funcsigs. 2020-06-03T18:45:12.722 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../088-python-funcsigs_1.0.2-4_all.deb ... 2020-06-03T18:45:12.766 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-06-03T18:45:12.850 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tz. 2020-06-03T18:45:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-python3-tz_2018.3-2_all.deb ... 2020-06-03T18:45:12.909 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tz (2018.3-2) ... 2020-06-03T18:45:13.069 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pluggy. 2020-06-03T18:45:13.083 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../089-python-pluggy_0.6.0-1_all.deb ... 2020-06-03T18:45:13.127 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-06-03T18:45:13.211 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-matplotlib. 2020-06-03T18:45:13.228 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-06-03T18:45:13.278 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-03T18:45:13.429 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-py. 2020-06-03T18:45:13.444 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../090-python-py_1.5.2-1_all.deb ... 2020-06-03T18:45:13.488 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-py (1.5.2-1) ... 2020-06-03T18:45:13.850 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pytest. 2020-06-03T18:45:13.865 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../091-python-pytest_3.3.2-2_all.deb ... 2020-06-03T18:45:13.916 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-06-03T18:45:13.967 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-olefile. 2020-06-03T18:45:13.983 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-python3-olefile_0.45.1-1_all.deb ... 2020-06-03T18:45:14.025 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-06-03T18:45:14.261 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-attr. 2020-06-03T18:45:14.275 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../092-python3-attr_17.4.0-2_all.deb ... 2020-06-03T18:45:14.327 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-06-03T18:45:14.345 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2020-06-03T18:45:14.360 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-2_all.deb ... 2020-06-03T18:45:14.403 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-06-03T18:45:14.672 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cycler. 2020-06-03T18:45:14.689 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../093-python3-cycler_0.10.0-1_all.deb ... 2020-06-03T18:45:14.730 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-06-03T18:45:14.748 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pil:amd64. 2020-06-03T18:45:14.762 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-06-03T18:45:14.806 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-03T18:45:15.025 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webencodings. 2020-06-03T18:45:15.040 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../094-python3-webencodings_0.5-2_all.deb ... 2020-06-03T18:45:15.083 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-06-03T18:45:15.210 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2020-06-03T18:45:15.225 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-python3-pluggy_0.6.0-1_all.deb ... 2020-06-03T18:45:15.268 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-06-03T18:45:15.369 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-html5lib. 2020-06-03T18:45:15.386 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../095-python3-html5lib_0.999999999-1_all.deb ... 2020-06-03T18:45:15.427 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-06-03T18:45:15.602 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2020-06-03T18:45:15.618 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-python3-py_1.5.2-1_all.deb ... 2020-06-03T18:45:15.668 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.5.2-1) ... 2020-06-03T18:45:15.789 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-06-03T18:45:15.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../096-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-06-03T18:45:15.847 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-03T18:45:16.005 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2020-06-03T18:45:16.020 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python3-pyinotify_0.9.6-1_all.deb ... 2020-06-03T18:45:16.063 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-06-03T18:45:16.275 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyparsing. 2020-06-03T18:45:16.290 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../097-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-06-03T18:45:16.358 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-03T18:45:16.391 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2020-06-03T18:45:16.405 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-pytest_3.3.2-2_all.deb ... 2020-06-03T18:45:16.449 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-06-03T18:45:16.686 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tz. 2020-06-03T18:45:16.701 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../098-python3-tz_2018.3-2_all.deb ... 2020-06-03T18:45:16.744 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tz (2018.3-2) ... 2020-06-03T18:45:16.803 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2020-06-03T18:45:16.819 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-python3-simplejson_3.13.2-1_amd64.deb ... 2020-06-03T18:45:16.861 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-06-03T18:45:17.047 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-matplotlib. 2020-06-03T18:45:17.065 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../099-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-06-03T18:45:17.105 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-03T18:45:17.198 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tk:amd64. 2020-06-03T18:45:17.212 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-06-03T18:45:17.256 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-03T18:45:17.743 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2020-06-03T18:45:17.759 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-radosgw_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:17.770 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-olefile. 2020-06-03T18:45:17.787 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../100-python3-olefile_0.45.1-1_all.deb ... 2020-06-03T18:45:17.801 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:17.828 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-06-03T18:45:18.130 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastescript. 2020-06-03T18:45:18.145 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-2_all.deb ... 2020-06-03T18:45:18.214 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-06-03T18:45:18.559 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pil:amd64. 2020-06-03T18:45:18.574 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../102-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-06-03T18:45:18.618 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-03T18:45:18.763 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2020-06-03T18:45:18.781 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../110-rbd-fuse_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:18.821 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:18.988 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pluggy. 2020-06-03T18:45:19.002 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../103-python3-pluggy_0.6.0-1_all.deb ... 2020-06-03T18:45:19.046 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-06-03T18:45:19.135 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-06-03T18:45:19.337 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-py. 2020-06-03T18:45:19.354 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../104-python3-py_1.5.2-1_all.deb ... 2020-06-03T18:45:19.396 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-py (1.5.2-1) ... 2020-06-03T18:45:19.454 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-03T18:45:19.716 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyinotify. 2020-06-03T18:45:19.730 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../105-python3-pyinotify_0.9.6-1_all.deb ... 2020-06-03T18:45:19.774 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-06-03T18:45:19.824 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-03T18:45:19.950 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-03T18:45:20.119 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pytest. 2020-06-03T18:45:20.133 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../106-python3-pytest_3.3.2-2_all.deb ... 2020-06-03T18:45:20.177 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-06-03T18:45:20.497 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplejson. 2020-06-03T18:45:20.513 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../107-python3-simplejson_3.13.2-1_amd64.deb ... 2020-06-03T18:45:20.555 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-06-03T18:45:20.569 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-03T18:45:20.884 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tk:amd64. 2020-06-03T18:45:20.898 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../108-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-06-03T18:45:20.951 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-03T18:45:20.962 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-06-03T18:45:21.246 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webencodings (0.5-2) ... 2020-06-03T18:45:21.404 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package radosgw. 2020-06-03T18:45:21.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../109-radosgw_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:21.462 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking radosgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:21.548 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-attr (17.4.0-2) ... 2020-06-03T18:45:21.741 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-06-03T18:45:21.867 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.5.2-1) ... 2020-06-03T18:45:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-06-03T18:45:22.391 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package rbd-fuse. 2020-06-03T18:45:22.405 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../110-rbd-fuse_16.0.0-2021-g230b0bb-1bionic_amd64.deb ... 2020-06-03T18:45:22.449 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking rbd-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:22.582 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-06-03T18:45:22.742 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-06-03T18:45:22.905 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-06-03T18:45:23.047 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-03T18:45:23.206 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:23.416 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-03T18:45:23.515 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-06-03T18:45:23.541 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-03T18:45:23.825 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-06-03T18:45:23.976 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-03T18:45:24.161 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-03T18:45:24.596 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-06-03T18:45:24.880 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webencodings (0.5-2) ... 2020-06-03T18:45:24.913 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-06-03T18:45:25.181 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-attr (17.4.0-2) ... 2020-06-03T18:45:25.257 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-06-03T18:45:25.382 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-06-03T18:45:25.509 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-py (1.5.2-1) ... 2020-06-03T18:45:25.550 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-py (1.5.2-1) ... 2020-06-03T18:45:25.776 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-06-03T18:45:25.860 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-06-03T18:45:26.116 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-06-03T18:45:26.214 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-06-03T18:45:26.214 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-06-03T18:45:26.502 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-06-03T18:45:26.538 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-06-03T18:45:26.622 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-06-03T18:45:26.823 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-06-03T18:45:26.839 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-argparse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:27.136 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-03T18:45:27.149 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-06-03T18:45:27.450 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-06-03T18:45:27.458 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-06-03T18:45:27.618 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-03T18:45:27.760 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-06-03T18:45:28.128 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-06-03T18:45:28.454 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-06-03T18:45:28.568 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-06-03T18:45:28.739 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:28.882 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-06-03T18:45:29.183 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-py (1.5.2-1) ... 2020-06-03T18:45:29.418 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-06-03T18:45:29.449 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2020-06-03T18:45:29.604 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-06-03T18:45:29.769 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-06-03T18:45:29.858 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-06-03T18:45:29.913 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-03T18:45:30.039 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-06-03T18:45:30.104 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-06-03T18:45:30.230 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-06-03T18:45:30.399 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-06-03T18:45:30.431 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-06-03T18:45:30.700 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-06-03T18:45:30.724 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-03T18:45:30.828 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-06-03T18:45:31.059 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-06-03T18:45:31.186 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:31.362 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-06-03T18:45:31.513 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-monotonic (1.1-2) ... 2020-06-03T18:45:31.753 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-06-03T18:45:31.841 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-03T18:45:31.974 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-06-03T18:45:32.090 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-06-03T18:45:32.333 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-03T18:45:32.397 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:32.423 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-06-03T18:45:32.463 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-06-03T18:45:32.710 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-06-03T18:45:33.020 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-03T18:45:33.069 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user cephadm....done 2020-06-03T18:45:33.253 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-06-03T18:45:33.312 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-03T18:45:33.555 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-03T18:45:33.647 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-attr (17.4.0-2) ... 2020-06-03T18:45:33.684 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-06-03T18:45:33.949 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-06-03T18:45:34.024 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-06-03T18:45:34.149 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-03T18:45:34.308 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-06-03T18:45:34.443 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-06-03T18:45:34.569 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-06-03T18:45:34.662 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-06-03T18:45:34.802 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:34.903 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0 (2.6.1-1) ... 2020-06-03T18:45:35.080 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-03T18:45:35.121 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-monotonic (1.1-2) ... 2020-06-03T18:45:35.230 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-03T18:45:35.422 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-03T18:45:35.490 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-06-03T18:45:35.548 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-06-03T18:45:35.616 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-06-03T18:45:35.916 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-03T18:45:36.001 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-03T18:45:36.006 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-06-03T18:45:36.046 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-06-03T18:45:36.127 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-06-03T18:45:36.285 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-06-03T18:45:36.420 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tz (2018.3-2) ... 2020-06-03T18:45:36.594 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-03T18:45:36.737 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-06-03T18:45:36.912 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-03T18:45:37.039 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-06-03T18:45:37.246 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-attr (17.4.0-2) ... 2020-06-03T18:45:37.341 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.11-1) ... 2020-06-03T18:45:37.548 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-06-03T18:45:37.716 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.4.1-1) ... 2020-06-03T18:45:37.740 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-03T18:45:38.026 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-06-03T18:45:38.352 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-06-03T18:45:38.392 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-06-03T18:45:38.482 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-06-03T18:45:38.670 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-03T18:45:38.752 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liboath0 (2.6.1-1) ... 2020-06-03T18:45:38.937 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-03T18:45:39.040 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-06-03T18:45:39.063 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-03T18:45:39.306 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-06-03T18:45:39.357 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.5+dfsg-2) ... 2020-06-03T18:45:39.432 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-06-03T18:45:39.483 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-06-03T18:45:39.817 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-03T18:45:39.953 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-06-03T18:45:40.262 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tz (2018.3-2) ... 2020-06-03T18:45:40.547 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-06-03T18:45:40.646 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-03T18:45:40.852 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-06-03T18:45:40.964 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-06-03T18:45:41.149 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-joblib (0.11-1) ... 2020-06-03T18:45:41.283 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-06-03T18:45:41.409 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-06-03T18:45:41.518 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-routes (2.4.1-1) ... 2020-06-03T18:45:41.835 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-06-03T18:45:42.135 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pytest (3.3.2-2) ... 2020-06-03T18:45:42.153 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-06-03T18:45:42.395 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-06-03T18:45:42.496 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-03T18:45:42.521 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-06-03T18:45:42.655 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:42.798 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-06-03T18:45:42.866 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-06-03T18:45:43.107 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:43.175 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up jq (1.5+dfsg-2) ... 2020-06-03T18:45:43.377 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:43.377 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:43.377 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-06-03T18:45:43.486 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-06-03T18:45:43.829 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-06-03T18:45:44.133 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-06-03T18:45:44.485 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-03T18:45:44.506 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-03T18:45:44.824 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-06-03T18:45:45.158 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-06-03T18:45:45.284 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-06-03T18:45:45.335 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-06-03T18:45:45.669 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pytest (3.3.2-2) ... 2020-06-03T18:45:45.964 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-06-03T18:45:46.118 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-06-03T18:45:46.182 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-06-03T18:45:46.256 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librados2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:46.382 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-06-03T18:45:46.568 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:46.700 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:46.712 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:46.826 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs-dev (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:46.952 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rados (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:47.012 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:47.087 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-06-03T18:45:47.138 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:47.373 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-03T18:45:47.373 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-03T18:45:47.413 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-06-03T18:45:47.674 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:47.723 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-06-03T18:45:47.806 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-06-03T18:45:47.889 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-06-03T18:45:47.930 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-06-03T18:45:48.066 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-03T18:45:48.244 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-03T18:45:48.897 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:48.936 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-06-03T18:45:49.031 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-06-03T18:45:49.366 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:49.492 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:49.618 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:49.745 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:49.821 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-06-03T18:45:50.210 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libradosstriper1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:50.289 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2020-06-03T18:45:50.344 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cephfs (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:50.646 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librgw2 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:50.691 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2020-06-03T18:45:50.780 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:50.875 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2020-06-03T18:45:51.005 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-03T18:45:51.006 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-03T18:45:51.034 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-06-03T18:45:51.275 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-06-03T18:45:51.307 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librbd1 (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:51.433 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-06-03T18:45:51.522 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-06-03T18:45:51.555 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-06-03T18:45:51.590 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:51.858 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-03T18:45:51.859 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-03T18:45:51.878 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-03T18:45:52.276 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:52.401 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:52.572 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-modules-core (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:52.698 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-06-03T18:45:52.735 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-06-03T18:45:53.024 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.037 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.151 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up rbd-fuse (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.245 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-03T18:45:53.245 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-03T18:45:53.276 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rbd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.444 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-common (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.563 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:53.714 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:54.020 INFO:teuthology.orchestra.run.smithi061.stdout:Adding group ceph....done 2020-06-03T18:45:54.186 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-03T18:45:54.186 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-03T18:45:54.424 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user ceph....done 2020-06-03T18:45:54.477 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:54.617 INFO:teuthology.orchestra.run.smithi061.stdout:Setting system user ceph properties....done 2020-06-03T18:45:54.619 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:54.761 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-rook (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:54.774 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-06-03T18:45:54.904 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:55.030 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-06-03T18:45:55.122 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-03T18:45:55.122 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-03T18:45:55.314 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up radosgw (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:55.413 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:55.604 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-03T18:45:55.605 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-03T18:45:55.626 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-03T18:45:55.626 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-03T18:45:55.900 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.000 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-test (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.026 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.126 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-base (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.168 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.295 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for systemd (237-3ubuntu10.39) ... 2020-06-03T18:45:56.465 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-06-03T18:45:56.488 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-06-03T18:45:56.719 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:56.952 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-03T18:45:56.952 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-03T18:45:57.274 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:57.422 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-osd (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:57.834 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-06-03T18:45:57.876 INFO:teuthology.orchestra.run.smithi112.stdout:ureadahead will be reprofiled on next reboot 2020-06-03T18:45:57.887 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-03T18:45:57.887 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-03T18:45:57.926 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-06-03T18:45:58.149 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-cephadm (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:58.293 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:58.402 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-06-03T18:45:58.405 DEBUG:teuthology.parallel:result is None 2020-06-03T18:45:58.435 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-rook (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:58.796 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mds (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:59.032 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-03T18:45:59.033 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-03T18:45:59.323 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mon (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:59.542 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-03T18:45:59.543 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-03T18:45:59.800 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-k8sevents (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:45:59.926 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-dashboard (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:46:00.069 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph (16.0.0-2021-g230b0bb-1bionic) ... 2020-06-03T18:46:00.204 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for systemd (237-3ubuntu10.39) ... 2020-06-03T18:46:00.397 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-06-03T18:46:01.807 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-06-03T18:46:01.850 INFO:teuthology.orchestra.run.smithi061.stdout:ureadahead will be reprofiled on next reboot 2020-06-03T18:46:01.909 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-06-03T18:46:02.365 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-06-03T18:46:02.367 DEBUG:teuthology.parallel:result is None 2020-06-03T18:46:02.368 INFO:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2020-06-03T18:46:02.425 INFO:teuthology.orchestra.run.smithi112.stdout:Ubuntu 2020-06-03T18:46:02.431 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:46:02.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=230b0bb1973ab49f8bd55921d5cad7dec0204c35 2020-06-03T18:46:02.641 INFO:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2020-06-03T18:46:02.666 INFO:teuthology.orchestra.run.smithi112.stdout:16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:46:02.668 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:46:02.668 INFO:teuthology.task.install:The correct ceph version 16.0.0-2021-g230b0bb-1bionic is installed. 2020-06-03T18:46:02.669 INFO:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2020-06-03T18:46:02.726 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2020-06-03T18:46:02.731 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-03T18:46:02.732 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=230b0bb1973ab49f8bd55921d5cad7dec0204c35 2020-06-03T18:46:02.936 INFO:teuthology.orchestra.run.smithi061:> dpkg-query -W -f '${Version}' ceph 2020-06-03T18:46:02.961 INFO:teuthology.orchestra.run.smithi061.stdout:16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:46:02.961 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-2021-g230b0bb-1bionic 2020-06-03T18:46:02.962 INFO:teuthology.task.install:The correct ceph version 16.0.0-2021-g230b0bb-1bionic is installed. 2020-06-03T18:46:02.962 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-06-03T18:46:02.964 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-03T18:46:02.994 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-03T18:46:03.017 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-06-03T18:46:03.018 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-03T18:46:03.050 INFO:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-03T18:46:03.107 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-03T18:46:03.124 INFO:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-03T18:46:03.184 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-06-03T18:46:03.185 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-03T18:46:03.239 INFO:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-03T18:46:03.293 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-03T18:46:03.340 INFO:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-03T18:46:03.404 INFO:teuthology.run_tasks:Running task ceph... 2020-06-03T18:46:03.511 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-06-03T18:46:03.512 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 777 /var/log/ceph 2020-06-03T18:46:03.518 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2020-06-03T18:46:03.540 INFO:tasks.ceph:Disabling ceph logrotate... 2020-06-03T18:46:03.541 INFO:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-03T18:46:03.592 INFO:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-03T18:46:03.617 INFO:tasks.ceph:Creating extra log directories... 2020-06-03T18:46:03.618 INFO:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-03T18:46:03.663 INFO:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-03T18:46:03.685 INFO:tasks.ceph:Creating ceph cluster c1... 2020-06-03T18:46:03.686 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-06-03T18:46:03.686 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016', 'branch': 'master', 'description': 'rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'email': None, 'first_in_suite': False, 'job_id': '5116016', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'fix_issue_43536', 'sha1': '390027420862af043e1952c99341e4057b1eb4e0'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'fix_issue_43536', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_fix_issue_43536/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '390027420862af043e1952c99341e4057b1eb4e0', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.8436', 'targets': {'smithi112.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/nBEuRLx53Ty/6d9ONoTJd876aJq75fLqp/5Y+tmafsmNhROmBI6pG9rUzCKDGOM2iueSI4SFXe4kV2VTqfftCY4vrWRdK4Xy0gZVyh+6pRaH5SbIDei5TAaN1M16VHvn/QJ4dYBxl1Wh4PpmlSFZVA37UVHaOHmHdVVUNdxteqrihmljgDL6K5FBsxncLzUzmVvnDcVKvcJnB0aq3n8DCzvswlf2fwQm6lJ7VO5XPxsa4n3PZeh/YIcAyyNZyADQUeBHrOfl2Zv2hFoQuODijEcyUTSMYkwvuZZ+2ILyGbpfmK1m6z91t62jgnU1PAg2OAtnRhbdM6ZHb+45DEKr', 'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiCUn2qnQjdIV5jntilANWO2UBbEruuFZKvHh00cy0SBueJd7dfgIIHf+B/h5JIO0ikBsLwdmTbIbbiZfXZx1lJvxHZWqe9yTUxej/ILchRWjVs30LngjD7lOwmoHrucD49q4sOcg8mDNLvyR+IQF8WvmGqeGEurxE34fX0KaEPlQgT1ivivdM3iXC71lELrZ+66UdcMUI766Buh3ZsiI3b7BwiaU14boJqHU4L67I5S0s+ibfGMTXAbqoCjfSHvBrcE+t2tx9D6In+E9C6cp16f1Avgi8uoZY5frwkY7D24EvOFIF4Xwu/VAcr30m83iiYgvjr9Y6Muii4pljWRsr'}} 2020-06-03T18:46:03.687 INFO:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-06-03T18:46:03.736 INFO:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-06-03T18:46:03.748 INFO:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2020-06-03T18:46:03.793 INFO:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2020-06-03T18:46:03.906 DEBUG:teuthology.orchestra.remote:smithi112:/scratch_devs is 72B 2020-06-03T18:46:03.946 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-03T18:46:03.947 INFO:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-03T18:46:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-03T18:46:03.962 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:03.962 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-06-03T18:46:03.962 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:03.963 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2020-06-03 18:43:39.351055585 +0000 2020-06-03T18:46:03.963 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2020-06-03 18:43:39.347053585 +0000 2020-06-03T18:46:03.963 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2020-06-03 18:43:39.347053585 +0000 2020-06-03T18:46:03.963 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2020-06-03T18:46:03.971 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2020-06-03T18:46:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2020-06-03T18:46:03.972 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000389787 s, 1.3 MB/s 2020-06-03T18:46:03.976 INFO:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-03T18:46:04.034 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-03T18:46:04.034 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:04.035 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-06-03T18:46:04.035 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:04.035 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2020-06-03 18:43:39.679219578 +0000 2020-06-03T18:46:04.035 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2020-06-03 18:43:39.563161581 +0000 2020-06-03T18:46:04.035 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2020-06-03 18:43:39.563161581 +0000 2020-06-03T18:46:04.036 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2020-06-03T18:46:04.044 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2020-06-03T18:46:04.045 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2020-06-03T18:46:04.045 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.00035558 s, 1.4 MB/s 2020-06-03T18:46:04.049 INFO:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-03T18:46:04.099 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-03T18:46:04.100 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:04.100 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-06-03T18:46:04.101 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:04.101 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2020-06-03 18:43:39.939349572 +0000 2020-06-03T18:46:04.101 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2020-06-03 18:43:39.815287575 +0000 2020-06-03T18:46:04.101 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2020-06-03 18:43:39.815287575 +0000 2020-06-03T18:46:04.101 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2020-06-03T18:46:04.108 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2020-06-03T18:46:04.109 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2020-06-03T18:46:04.109 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000326944 s, 1.6 MB/s 2020-06-03T18:46:04.115 INFO:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-03T18:46:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-03T18:46:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:04.164 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-06-03T18:46:04.165 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:04.165 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2020-06-03 18:43:40.191475567 +0000 2020-06-03T18:46:04.165 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2020-06-03 18:43:40.075417569 +0000 2020-06-03T18:46:04.165 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2020-06-03 18:43:40.075417569 +0000 2020-06-03T18:46:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2020-06-03T18:46:04.173 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2020-06-03T18:46:04.174 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2020-06-03T18:46:04.174 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000325078 s, 1.6 MB/s 2020-06-03T18:46:04.178 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:04.179 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-03T18:46:04.179 INFO:tasks.ceph:Generating config... 2020-06-03T18:46:04.183 INFO:tasks.ceph:[client] debug rgw = 20 2020-06-03T18:46:04.183 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-06-03T18:46:04.184 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-06-03T18:46:04.185 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:04.185 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-06-03T18:46:04.185 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-06-03T18:46:04.186 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-06-03T18:46:04.186 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-06-03T18:46:04.186 INFO:tasks.ceph:[client] setgroup = ceph 2020-06-03T18:46:04.187 INFO:tasks.ceph:[client] setuser = ceph 2020-06-03T18:46:04.187 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-06-03T18:46:04.187 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-06-03T18:46:04.187 INFO:tasks.ceph:[mgr] debug ms = 1 2020-06-03T18:46:04.188 INFO:tasks.ceph:[mon] debug mon = 20 2020-06-03T18:46:04.188 INFO:tasks.ceph:[mon] debug ms = 1 2020-06-03T18:46:04.188 INFO:tasks.ceph:[mon] debug paxos = 20 2020-06-03T18:46:04.189 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-06-03T18:46:04.189 INFO:tasks.ceph:[osd] debug filestore = 20 2020-06-03T18:46:04.189 INFO:tasks.ceph:[osd] debug journal = 20 2020-06-03T18:46:04.190 INFO:tasks.ceph:[osd] debug ms = 20 2020-06-03T18:46:04.190 INFO:tasks.ceph:[osd] debug osd = 25 2020-06-03T18:46:04.191 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-06-03T18:46:04.191 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-06-03T18:46:04.193 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-06-03T18:46:04.193 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:04.194 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:04.214 INFO:tasks.ceph:Setting up c1.mon.a... 2020-06-03T18:46:04.215 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-06-03T18:46:04.257 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.keyring 2020-06-03T18:46:04.260 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-06-03T18:46:04.338 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-03T18:46:04.396 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.112')] 2020-06-03T18:46:04.397 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.112 --print /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:04.468 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:04.468 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid 308e03be-58c9-4f9c-8b62-81277e23a24f 2020-06-03T18:46:04.469 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2020-06-03T18:46:04.469 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 308e03be-58c9-4f9c-8b62-81277e23a24f 2020-06-03T18:46:04.469 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2020-06-03T18:46:04.469406+0000 2020-06-03T18:46:04.469 INFO:teuthology.orchestra.run.smithi112.stdout:created 2020-06-03T18:46:04.469406+0000 2020-06-03T18:46:04.469 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 0 (unknown) 2020-06-03T18:46:04.470 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2020-06-03T18:46:04.470 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-06-03T18:46:04.471 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 308e03be-58c9-4f9c-8b62-81277e23a24f... 2020-06-03T18:46:04.472 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-06-03T18:46:04.476 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-06-03T18:46:04.516 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2020-06-03T18:46:04.516 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2020-06-03T18:46:04.516 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-03T18:46:04.517 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2020-06-03T18:46:04.517 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.517 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2020-06-03T18:46:04.517 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.517 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2020-06-03T18:46:04.518 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.518 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2020-06-03T18:46:04.518 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2020-06-03T18:46:04.518 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.518 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.519 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2020-06-03T18:46:04.520 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-03T18:46:04.520 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.520 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2020-06-03T18:46:04.520 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.520 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2020-06-03T18:46:04.521 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2020-06-03T18:46:04.522 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-03T18:46:04.522 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-03T18:46:04.522 INFO:teuthology.orchestra.run.smithi061.stdout: mon_allow_pool_size_one = true 2020-06-03T18:46:04.522 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.522 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.523 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2020-06-03T18:46:04.524 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2020-06-03T18:46:04.524 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = 172.21.15.112 2020-06-03T18:46:04.524 INFO:teuthology.orchestra.run.smithi061.stdout: osd heartbeat grace = 40 2020-06-03T18:46:04.524 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 308e03be-58c9-4f9c-8b62-81277e23a24f 2020-06-03T18:46:04.524 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.525 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2020-06-03T18:46:04.525 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2020-06-03T18:46:04.525 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.525 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2020-06-03T18:46:04.525 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2020-06-03T18:46:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2020-06-03T18:46:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2020-06-03T18:46:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2020-06-03T18:46:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2020-06-03T18:46:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2020-06-03T18:46:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2020-06-03T18:46:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2020-06-03T18:46:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2020-06-03T18:46:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2020-06-03T18:46:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 20 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 20 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 20 2020-06-03T18:46:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 25 2020-06-03T18:46:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: osd fast shutdown = False 2020-06-03T18:46:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: rocksdb delete range threshold = 0 2020-06-03T18:46:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.531 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2020-06-03T18:46:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2020-06-03T18:46:04.533 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.533 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2020-06-03T18:46:04.539 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2020-06-03T18:46:04.540 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2020-06-03T18:46:04.540 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2020-06-03T18:46:04.540 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2020-06-03T18:46:04.540 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2020-06-03T18:46:04.540 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2020-06-03T18:46:04.541 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:04.541 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2020-06-03T18:46:04.541 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2020-06-03T18:46:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd crush smoke test = False 2020-06-03T18:46:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-03T18:46:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-03T18:46:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw = 20 2020-06-03T18:46:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt require ssl = False 2020-06-03T18:46:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms backend = testing 2020-06-03T18:46:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: rgw curl low speed time = 300 2020-06-03T18:46:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: rgw data log num shards = 4 2020-06-03T18:46:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: rgw md log max shards = 4 2020-06-03T18:46:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync log trim interval = 0 2020-06-03T18:46:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: setgroup = ceph 2020-06-03T18:46:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: setuser = ceph 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:04.554 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2020-06-03T18:46:04.555 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:04.556 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2020-06-03T18:46:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2020-06-03T18:46:04.556 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-03T18:46:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2020-06-03T18:46:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2020-06-03T18:46:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.557 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2020-06-03T18:46:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2020-06-03T18:46:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2020-06-03T18:46:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.558 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.559 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2020-06-03T18:46:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-03T18:46:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2020-06-03T18:46:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.560 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2020-06-03T18:46:04.561 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2020-06-03T18:46:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-03T18:46:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-03T18:46:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2020-06-03T18:46:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.562 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-03T18:46:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2020-06-03T18:46:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.563 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2020-06-03T18:46:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2020-06-03T18:46:04.564 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2020-06-03T18:46:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.112 2020-06-03T18:46:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: osd heartbeat grace = 40 2020-06-03T18:46:04.565 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 308e03be-58c9-4f9c-8b62-81277e23a24f 2020-06-03T18:46:04.566 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.567 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2020-06-03T18:46:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2020-06-03T18:46:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2020-06-03T18:46:04.567 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2020-06-03T18:46:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2020-06-03T18:46:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2020-06-03T18:46:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.568 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2020-06-03T18:46:04.569 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2020-06-03T18:46:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2020-06-03T18:46:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.570 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2020-06-03T18:46:04.571 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2020-06-03T18:46:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: debug filestore = 20 2020-06-03T18:46:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: debug journal = 20 2020-06-03T18:46:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 20 2020-06-03T18:46:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 25 2020-06-03T18:46:04.572 INFO:teuthology.orchestra.run.smithi112.stdout: osd fast shutdown = False 2020-06-03T18:46:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2020-06-03T18:46:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.573 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2020-06-03T18:46:04.573 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:04.574 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2020-06-03T18:46:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.575 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2020-06-03T18:46:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2020-06-03T18:46:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2020-06-03T18:46:04.575 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2020-06-03T18:46:04.576 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2020-06-03T18:46:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd crush smoke test = False 2020-06-03T18:46:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:04.577 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2020-06-03T18:46:04.577 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2020-06-03T18:46:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2020-06-03T18:46:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2020-06-03T18:46:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-03T18:46:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-03T18:46:04.578 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2020-06-03T18:46:04.579 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2020-06-03T18:46:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2020-06-03T18:46:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2020-06-03T18:46:04.580 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2020-06-03T18:46:04.580 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2020-06-03T18:46:04.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2020-06-03T18:46:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2020-06-03T18:46:04.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2020-06-03T18:46:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:04.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2020-06-03T18:46:04.581 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:04.582 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-06-03T18:46:04.582 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-06-03T18:46:04.647 INFO:tasks.ceph:Copying monmap to all nodes... 2020-06-03T18:46:04.700 DEBUG:teuthology.orchestra.remote:smithi112:/etc/ceph/c1.keyring is 206B 2020-06-03T18:46:04.729 DEBUG:teuthology.orchestra.remote:smithi112:/home/ubuntu/cephtest/c1.monmap is 179B 2020-06-03T18:46:04.744 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:46:04.744 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-03T18:46:04.800 INFO:teuthology.orchestra.run.smithi112:> cat > /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:04.890 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:46:04.891 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-03T18:46:04.951 INFO:teuthology.orchestra.run.smithi061:> cat > /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:05.042 INFO:tasks.ceph:Setting up mon nodes... 2020-06-03T18:46:05.043 INFO:tasks.ceph:Setting up mgr nodes... 2020-06-03T18:46:05.043 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-06-03T18:46:05.086 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-06-03T18:46:05.088 INFO:tasks.ceph:Setting up mds nodes... 2020-06-03T18:46:05.089 INFO:tasks.ceph_client:Setting up client nodes... 2020-06-03T18:46:05.090 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-06-03T18:46:05.165 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.0.keyring 2020-06-03T18:46:05.177 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-06-03T18:46:05.252 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.1.keyring 2020-06-03T18:46:05.264 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-06-03T18:46:05.264 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-03T18:46:05.265 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-06-03T18:46:05.318 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:05.318 INFO:tasks.ceph:role: c1.osd.0 2020-06-03T18:46:05.319 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:46:05.319 INFO:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-03T18:46:05.842 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:05.844 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:05.845 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:05.845 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:05.846 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:05.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2020-06-03T18:46:05.848 INFO:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-06-03T18:46:05.934 INFO:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-06-03T18:46:05.944 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:05.946 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-06-03T18:46:06.003 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:06.004 INFO:tasks.ceph:role: c1.osd.1 2020-06-03T18:46:06.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:46:06.005 INFO:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-03T18:46:06.516 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:06.517 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:06.517 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:06.517 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:06.518 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:06.518 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:06.518 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:06.518 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:06.518 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:06.519 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2020-06-03T18:46:06.520 INFO:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-06-03T18:46:06.584 INFO:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-06-03T18:46:06.599 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:06.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:06.600 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-06-03T18:46:06.660 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:06.660 INFO:tasks.ceph:role: c1.osd.2 2020-06-03T18:46:06.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:46:06.661 INFO:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-03T18:46:07.159 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:07.161 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:07.161 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:07.161 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:07.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2020-06-03T18:46:07.163 INFO:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-06-03T18:46:07.219 INFO:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-06-03T18:46:07.230 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:07.232 INFO:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:07.326 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.326+0000 7f3da43bad80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-06-03T18:46:07.326 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.326+0000 7f3da43bad80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-06-03T18:46:07.326 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.326+0000 7f3da43bad80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-03T18:46:07.327 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.326+0000 7f3da43bad80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-03T18:46:07.327 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.326+0000 7f3da43bad80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-03T18:46:07.338 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:07.330+0000 7f3da43bad80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-06-03T18:46:09.252 INFO:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-06-03T18:46:09.269 INFO:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:09.350 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-06-03T18:46:09.350 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-06-03T18:46:09.351 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-03T18:46:09.351 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-03T18:46:09.351 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-03T18:46:09.351 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:09.350+0000 7f52bab18d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-06-03T18:46:11.284 INFO:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-06-03T18:46:11.343 INFO:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:11.423 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-06-03T18:46:11.423 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-06-03T18:46:11.423 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-03T18:46:11.423 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-03T18:46:11.424 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-03T18:46:11.424 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:46:11.422+0000 7f617983fd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-06-03T18:46:13.358 INFO:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-06-03T18:46:13.420 INFO:tasks.ceph:Reading keys from all nodes... 2020-06-03T18:46:13.421 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:13.470 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.z8uBzkYxIO 2020-06-03T18:46:13.471 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.z8uBzkYxIO 2020-06-03T18:46:13.527 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.z8uBzkYxIO 2020-06-03T18:46:13.637 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.z8uBzkYxIO is 56B 2020-06-03T18:46:13.690 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.z8uBzkYxIO 2020-06-03T18:46:13.696 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:13.745 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.NOkKR4lWRH 2020-06-03T18:46:13.746 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.NOkKR4lWRH 2020-06-03T18:46:13.803 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.NOkKR4lWRH 2020-06-03T18:46:13.916 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.NOkKR4lWRH is 56B 2020-06-03T18:46:13.927 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.NOkKR4lWRH 2020-06-03T18:46:13.934 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:13.982 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.Q9DOSyLzAP 2020-06-03T18:46:13.982 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.Q9DOSyLzAP 2020-06-03T18:46:14.039 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.Q9DOSyLzAP 2020-06-03T18:46:14.145 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.Q9DOSyLzAP is 56B 2020-06-03T18:46:14.155 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.Q9DOSyLzAP 2020-06-03T18:46:14.161 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:14.209 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.CGhyhfGL7S 2020-06-03T18:46:14.245 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.CGhyhfGL7S 2020-06-03T18:46:14.275 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.CGhyhfGL7S 2020-06-03T18:46:14.403 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.CGhyhfGL7S is 56B 2020-06-03T18:46:14.420 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.CGhyhfGL7S 2020-06-03T18:46:14.483 DEBUG:teuthology.orchestra.remote:smithi112:/etc/ceph/c1.client.0.keyring is 59B 2020-06-03T18:46:14.542 DEBUG:teuthology.orchestra.remote:smithi112:/etc/ceph/c1.client.1.keyring is 59B 2020-06-03T18:46:14.553 INFO:tasks.ceph:Adding keys to all mons... 2020-06-03T18:46:14.554 INFO:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/c1.keyring 2020-06-03T18:46:14.575 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2020-06-03T18:46:14.576 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDt79deKOg7BRAA8WRwDzTbq+OiYYXHVAT9lw== 2020-06-03T18:46:14.576 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2020-06-03T18:46:14.576 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDv79dejoF6ExAAwG1cLzgvVM017InKdONeFA== 2020-06-03T18:46:14.576 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2020-06-03T18:46:14.576 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDx79dePgXwFBAAOC058ueJyBGLkniHJNEYXw== 2020-06-03T18:46:14.577 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2020-06-03T18:46:14.577 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDz79deaYZFGRAAYfn+l5TAFB4oQQZobwfsbQ== 2020-06-03T18:46:14.577 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2020-06-03T18:46:14.577 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDt79deNFTqCRAAtIDJ7rRm/BYwfHwpBEvlTw== 2020-06-03T18:46:14.578 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2020-06-03T18:46:14.578 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQDt79deDwsjDxAA6mC1DCPUQ1X0Sx71hOPg0A== 2020-06-03T18:46:14.579 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-03T18:46:14.662 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:14.746 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:14.830 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:14.916 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-03T18:46:15.008 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-03T18:46:15.098 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-06-03T18:46:15.099 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-06-03T18:46:15.112 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-06-03T18:46:15.206 INFO:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-06-03T18:46:15.221 INFO:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-06-03T18:46:15.269 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-06-03T18:46:15.270 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-06-03T18:46:15.271 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-06-03T18:46:15.317 INFO:tasks.ceph.c1.mon.a:Started 2020-06-03T18:46:15.317 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-06-03T18:46:15.318 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-06-03T18:46:15.318 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-06-03T18:46:15.320 INFO:tasks.ceph.c1.mgr.x:Started 2020-06-03T18:46:15.321 INFO:tasks.ceph:Setting crush tunables to default 2020-06-03T18:46:15.321 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd crush tunables default 2020-06-03T18:46:15.760 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2020-06-03T18:46:15.776 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-06-03T18:46:15.777 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:15.787 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.aa2vc8JrSR 2020-06-03T18:46:15.787 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.aa2vc8JrSR 2020-06-03T18:46:15.852 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.aa2vc8JrSR 2020-06-03T18:46:15.996 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.aa2vc8JrSR is 37B 2020-06-03T18:46:16.026 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.aa2vc8JrSR 2020-06-03T18:46:16.033 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:16.087 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.ZvTRTLOP3q 2020-06-03T18:46:16.088 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.ZvTRTLOP3q 2020-06-03T18:46:16.142 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.ZvTRTLOP3q 2020-06-03T18:46:16.248 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.ZvTRTLOP3q is 37B 2020-06-03T18:46:16.260 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.ZvTRTLOP3q 2020-06-03T18:46:16.274 INFO:teuthology.orchestra.run.smithi112:> mktemp 2020-06-03T18:46:16.322 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.PwAIpr4YWy 2020-06-03T18:46:16.323 INFO:teuthology.orchestra.run.smithi112:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.PwAIpr4YWy 2020-06-03T18:46:16.379 INFO:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.PwAIpr4YWy 2020-06-03T18:46:16.485 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.PwAIpr4YWy is 37B 2020-06-03T18:46:16.508 INFO:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.PwAIpr4YWy 2020-06-03T18:46:16.520 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new 2465bb39-363b-4c4b-8ede-ddfb867e26ea 0 2020-06-03T18:46:16.760 INFO:teuthology.orchestra.run.smithi112.stdout:0 2020-06-03T18:46:16.771 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new 0a712ead-11a6-4113-918d-bda5b944ec8d 1 2020-06-03T18:46:16.998 INFO:teuthology.orchestra.run.smithi112.stdout:1 2020-06-03T18:46:17.008 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7 2 2020-06-03T18:46:17.206 INFO:teuthology.orchestra.run.smithi112.stdout:2 2020-06-03T18:46:17.218 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-06-03T18:46:17.219 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-06-03T18:46:17.223 INFO:tasks.ceph.c1.osd.0:Started 2020-06-03T18:46:17.223 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-06-03T18:46:17.224 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-06-03T18:46:17.228 INFO:tasks.ceph.c1.osd.1:Started 2020-06-03T18:46:17.228 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-06-03T18:46:17.229 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-06-03T18:46:17.238 INFO:tasks.ceph.c1.osd.2:Started 2020-06-03T18:46:17.239 INFO:tasks.ceph:Waiting for OSDs to come up 2020-06-03T18:46:17.544 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-06-03T18:46:17.723 INFO:teuthology.misc.health.smithi112.stdout: 2020-06-03T18:46:17.724 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":5,"fsid":"308e03be-58c9-4f9c-8b62-81277e23a24f","created":"2020-06-03T18:46:15.427134+0000","modified":"2020-06-03T18:46:17.204883+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"2465bb39-363b-4c4b-8ede-ddfb867e26ea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0a712ead-11a6-4113-918d-bda5b944ec8d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:17.735 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-06-03T18:46:17.854 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2020-06-03T18:46:17.854+0000 7f7655d05d80 -1 Falling back to public interface 2020-06-03T18:46:17.858 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2020-06-03T18:46:17.858+0000 7fbd56661d80 -1 Falling back to public interface 2020-06-03T18:46:17.863 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2020-06-03T18:46:17.862+0000 7f5b572fad80 -1 Falling back to public interface 2020-06-03T18:46:18.474 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2020-06-03T18:46:18.474+0000 7f7655d05d80 -1 osd.1 0 log_to_monitors {default=true} 2020-06-03T18:46:18.482 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2020-06-03T18:46:18.482+0000 7fbd56661d80 -1 osd.2 0 log_to_monitors {default=true} 2020-06-03T18:46:18.484 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2020-06-03T18:46:18.486+0000 7f5b572fad80 -1 osd.0 0 log_to_monitors {default=true} 2020-06-03T18:46:20.463 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2020-06-03T18:46:20.462+0000 7fbd4db2c700 -1 osd.2 0 waiting for initial osdmap 2020-06-03T18:46:20.464 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2020-06-03T18:46:20.462+0000 7f764d1d0700 -1 osd.1 0 waiting for initial osdmap 2020-06-03T18:46:20.465 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2020-06-03T18:46:20.466+0000 7f5b4e7c5700 -1 osd.0 0 waiting for initial osdmap 2020-06-03T18:46:24.040 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-06-03T18:46:24.302 INFO:teuthology.misc.health.smithi112.stdout: 2020-06-03T18:46:24.302 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"308e03be-58c9-4f9c-8b62-81277e23a24f","created":"2020-06-03T18:46:15.427134+0000","modified":"2020-06-03T18:46:23.476975+0000","last_up_change":"2020-06-03T18:46:21.458888+0000","last_in_change":"2020-06-03T18:46:21.458888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:19.122635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2465bb39-363b-4c4b-8ede-ddfb867e26ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":13589},{"type":"v1","addr":"172.21.15.112:6817","nonce":13589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":13589},{"type":"v1","addr":"172.21.15.112:6819","nonce":13589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":13589},{"type":"v1","addr":"172.21.15.112:6823","nonce":13589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":13589},{"type":"v1","addr":"172.21.15.112:6821","nonce":13589}]},"public_addr":"172.21.15.112:6817/13589","cluster_addr":"172.21.15.112:6819/13589","heartbeat_back_addr":"172.21.15.112:6823/13589","heartbeat_front_addr":"172.21.15.112:6821/13589","state":["exists","up"]},{"osd":1,"uuid":"0a712ead-11a6-4113-918d-bda5b944ec8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":13590},{"type":"v1","addr":"172.21.15.112:6801","nonce":13590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":13590},{"type":"v1","addr":"172.21.15.112:6803","nonce":13590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":13590},{"type":"v1","addr":"172.21.15.112:6807","nonce":13590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":13590},{"type":"v1","addr":"172.21.15.112:6805","nonce":13590}]},"public_addr":"172.21.15.112:6801/13590","cluster_addr":"172.21.15.112:6803/13590","heartbeat_back_addr":"172.21.15.112:6807/13590","heartbeat_front_addr":"172.21.15.112:6805/13590","state":["exists","up"]},{"osd":2,"uuid":"cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":13591},{"type":"v1","addr":"172.21.15.112:6809","nonce":13591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":13591},{"type":"v1","addr":"172.21.15.112:6811","nonce":13591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":13591},{"type":"v1","addr":"172.21.15.112:6815","nonce":13591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":13591},{"type":"v1","addr":"172.21.15.112:6813","nonce":13591}]},"public_addr":"172.21.15.112:6809/13591","cluster_addr":"172.21.15.112:6811/13591","heartbeat_back_addr":"172.21.15.112:6815/13591","heartbeat_front_addr":"172.21.15.112:6813/13591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.518047+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.488336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.493702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:24.314 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-06-03T18:46:24.314 INFO:tasks.ceph:Creating RBD pool 2020-06-03T18:46:24.315 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool create rbd 8 2020-06-03T18:46:25.506 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2020-06-03T18:46:25.522 INFO:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-06-03T18:46:26.509 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rbd' on pool 'rbd' 2020-06-03T18:46:26.522 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-06-03T18:46:26.523 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-03T18:46:26.524 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-06-03T18:46:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:26.779 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":12,"fsid":"308e03be-58c9-4f9c-8b62-81277e23a24f","created":"2020-06-03T18:46:15.427134+0000","modified":"2020-06-03T18:46:26.492857+0000","last_up_change":"2020-06-03T18:46:21.458888+0000","last_in_change":"2020-06-03T18:46:21.458888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:19.122635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:24.572477+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2465bb39-363b-4c4b-8ede-ddfb867e26ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":13589},{"type":"v1","addr":"172.21.15.112:6817","nonce":13589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":13589},{"type":"v1","addr":"172.21.15.112:6819","nonce":13589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":13589},{"type":"v1","addr":"172.21.15.112:6823","nonce":13589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":13589},{"type":"v1","addr":"172.21.15.112:6821","nonce":13589}]},"public_addr":"172.21.15.112:6817/13589","cluster_addr":"172.21.15.112:6819/13589","heartbeat_back_addr":"172.21.15.112:6823/13589","heartbeat_front_addr":"172.21.15.112:6821/13589","state":["exists","up"]},{"osd":1,"uuid":"0a712ead-11a6-4113-918d-bda5b944ec8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":13590},{"type":"v1","addr":"172.21.15.112:6801","nonce":13590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":13590},{"type":"v1","addr":"172.21.15.112:6803","nonce":13590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":13590},{"type":"v1","addr":"172.21.15.112:6807","nonce":13590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":13590},{"type":"v1","addr":"172.21.15.112:6805","nonce":13590}]},"public_addr":"172.21.15.112:6801/13590","cluster_addr":"172.21.15.112:6803/13590","heartbeat_back_addr":"172.21.15.112:6807/13590","heartbeat_front_addr":"172.21.15.112:6805/13590","state":["exists","up"]},{"osd":2,"uuid":"cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":13591},{"type":"v1","addr":"172.21.15.112:6809","nonce":13591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":13591},{"type":"v1","addr":"172.21.15.112:6811","nonce":13591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":13591},{"type":"v1","addr":"172.21.15.112:6815","nonce":13591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":13591},{"type":"v1","addr":"172.21.15.112:6813","nonce":13591}]},"public_addr":"172.21.15.112:6809/13591","cluster_addr":"172.21.15.112:6811/13591","heartbeat_back_addr":"172.21.15.112:6815/13591","heartbeat_front_addr":"172.21.15.112:6813/13591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.518047+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.488336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.493702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:26.793 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-03T18:46:19.122635+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-03T18:46:24.572477+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-06-03T18:46:26.794 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-06-03T18:46:27.059 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2020-06-03T18:46:27.080 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-06-03T18:46:27.342 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2020-06-03T18:46:27.355 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-06-03T18:46:27.355 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-06-03T18:46:27.356 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-06-03T18:46:27.659 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:27.672 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":13386},{"type":"v1","addr":"172.21.15.112:6825","nonce":13386}]},"active_addr":"172.21.15.112:6825/13386","active_change":"2020-06-03T18:46:19.087977+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:latest","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4097153251}]},{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2483126302}]},{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":816281149}]}]}} 2020-06-03T18:46:27.675 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-06-03T18:46:27.675 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-06-03T18:46:27.676 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-03T18:46:27.926 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:27.927 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":13,"fsid":"308e03be-58c9-4f9c-8b62-81277e23a24f","created":"2020-06-03T18:46:15.427134+0000","modified":"2020-06-03T18:46:27.501507+0000","last_up_change":"2020-06-03T18:46:21.458888+0000","last_in_change":"2020-06-03T18:46:21.458888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:19.122635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:24.572477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2465bb39-363b-4c4b-8ede-ddfb867e26ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":13589},{"type":"v1","addr":"172.21.15.112:6817","nonce":13589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":13589},{"type":"v1","addr":"172.21.15.112:6819","nonce":13589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":13589},{"type":"v1","addr":"172.21.15.112:6823","nonce":13589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":13589},{"type":"v1","addr":"172.21.15.112:6821","nonce":13589}]},"public_addr":"172.21.15.112:6817/13589","cluster_addr":"172.21.15.112:6819/13589","heartbeat_back_addr":"172.21.15.112:6823/13589","heartbeat_front_addr":"172.21.15.112:6821/13589","state":["exists","up"]},{"osd":1,"uuid":"0a712ead-11a6-4113-918d-bda5b944ec8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":13590},{"type":"v1","addr":"172.21.15.112:6801","nonce":13590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":13590},{"type":"v1","addr":"172.21.15.112:6803","nonce":13590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":13590},{"type":"v1","addr":"172.21.15.112:6807","nonce":13590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":13590},{"type":"v1","addr":"172.21.15.112:6805","nonce":13590}]},"public_addr":"172.21.15.112:6801/13590","cluster_addr":"172.21.15.112:6803/13590","heartbeat_back_addr":"172.21.15.112:6807/13590","heartbeat_front_addr":"172.21.15.112:6805/13590","state":["exists","up"]},{"osd":2,"uuid":"cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":13591},{"type":"v1","addr":"172.21.15.112:6809","nonce":13591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":13591},{"type":"v1","addr":"172.21.15.112:6811","nonce":13591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":13591},{"type":"v1","addr":"172.21.15.112:6815","nonce":13591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":13591},{"type":"v1","addr":"172.21.15.112:6813","nonce":13591}]},"public_addr":"172.21.15.112:6809/13591","cluster_addr":"172.21.15.112:6811/13591","heartbeat_back_addr":"172.21.15.112:6815/13591","heartbeat_front_addr":"172.21.15.112:6813/13591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.518047+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.488336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.493702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:27.939 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-06-03T18:46:27.940 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-03T18:46:28.194 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:28.195 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":13,"fsid":"308e03be-58c9-4f9c-8b62-81277e23a24f","created":"2020-06-03T18:46:15.427134+0000","modified":"2020-06-03T18:46:27.501507+0000","last_up_change":"2020-06-03T18:46:21.458888+0000","last_in_change":"2020-06-03T18:46:21.458888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:19.122635+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:24.572477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2465bb39-363b-4c4b-8ede-ddfb867e26ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":13589},{"type":"v1","addr":"172.21.15.112:6817","nonce":13589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":13589},{"type":"v1","addr":"172.21.15.112:6819","nonce":13589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":13589},{"type":"v1","addr":"172.21.15.112:6823","nonce":13589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":13589},{"type":"v1","addr":"172.21.15.112:6821","nonce":13589}]},"public_addr":"172.21.15.112:6817/13589","cluster_addr":"172.21.15.112:6819/13589","heartbeat_back_addr":"172.21.15.112:6823/13589","heartbeat_front_addr":"172.21.15.112:6821/13589","state":["exists","up"]},{"osd":1,"uuid":"0a712ead-11a6-4113-918d-bda5b944ec8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":13590},{"type":"v1","addr":"172.21.15.112:6801","nonce":13590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":13590},{"type":"v1","addr":"172.21.15.112:6803","nonce":13590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":13590},{"type":"v1","addr":"172.21.15.112:6807","nonce":13590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":13590},{"type":"v1","addr":"172.21.15.112:6805","nonce":13590}]},"public_addr":"172.21.15.112:6801/13590","cluster_addr":"172.21.15.112:6803/13590","heartbeat_back_addr":"172.21.15.112:6807/13590","heartbeat_front_addr":"172.21.15.112:6805/13590","state":["exists","up"]},{"osd":2,"uuid":"cbe60ef5-3a7f-487e-82d7-4bb0c0997ae7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":13591},{"type":"v1","addr":"172.21.15.112:6809","nonce":13591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":13591},{"type":"v1","addr":"172.21.15.112:6811","nonce":13591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":13591},{"type":"v1","addr":"172.21.15.112:6815","nonce":13591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":13591},{"type":"v1","addr":"172.21.15.112:6813","nonce":13591}]},"public_addr":"172.21.15.112:6809/13591","cluster_addr":"172.21.15.112:6811/13591","heartbeat_back_addr":"172.21.15.112:6815/13591","heartbeat_front_addr":"172.21.15.112:6813/13591","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.518047+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.488336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:19.493702+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:28.205 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-06-03T18:46:28.357 INFO:teuthology.orchestra.run.smithi112.stdout:34359738373 2020-06-03T18:46:28.358 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-06-03T18:46:28.516 INFO:teuthology.orchestra.run.smithi112.stdout:34359738373 2020-06-03T18:46:28.517 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-06-03T18:46:28.677 INFO:teuthology.orchestra.run.smithi112.stdout:34359738372 2020-06-03T18:46:28.679 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-06-03T18:46:28.955 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2020-06-03T18:46:28.970 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738371 for osd.0 2020-06-03T18:46:29.972 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-06-03T18:46:30.242 INFO:teuthology.orchestra.run.smithi112.stdout:34359738373 2020-06-03T18:46:30.250 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.0 2020-06-03T18:46:30.250 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-06-03T18:46:30.520 INFO:teuthology.orchestra.run.smithi112.stdout:34359738373 2020-06-03T18:46:30.534 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738373 got 34359738373 for osd.1 2020-06-03T18:46:30.535 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-06-03T18:46:30.801 INFO:teuthology.orchestra.run.smithi112.stdout:34359738372 2020-06-03T18:46:30.814 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2020-06-03T18:46:30.814 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-06-03T18:46:30.815 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-06-03T18:46:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2020-06-03T18:46:31.090 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-03T18:46:29.098609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":93582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.592266"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513413+0000","last_change":"2020-06-03T18:46:26.513069+0000","last_active":"2020-06-03T18:46:26.513413+0000","last_peered":"2020-06-03T18:46:26.513413+0000","last_clean":"2020-06-03T18:46:26.513413+0000","last_became_active":"2020-06-03T18:46:26.512547+0000","last_became_peered":"2020-06-03T18:46:26.512547+0000","last_unstale":"2020-06-03T18:46:26.513413+0000","last_undegraded":"2020-06-03T18:46:26.513413+0000","last_fullsized":"2020-06-03T18:46:26.513413+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513294+0000","last_change":"2020-06-03T18:46:26.512733+0000","last_active":"2020-06-03T18:46:26.513294+0000","last_peered":"2020-06-03T18:46:26.513294+0000","last_clean":"2020-06-03T18:46:26.513294+0000","last_became_active":"2020-06-03T18:46:26.511498+0000","last_became_peered":"2020-06-03T18:46:26.511498+0000","last_unstale":"2020-06-03T18:46:26.513294+0000","last_undegraded":"2020-06-03T18:46:26.513294+0000","last_fullsized":"2020-06-03T18:46:26.513294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513333+0000","last_change":"2020-06-03T18:46:26.512686+0000","last_active":"2020-06-03T18:46:26.513333+0000","last_peered":"2020-06-03T18:46:26.513333+0000","last_clean":"2020-06-03T18:46:26.513333+0000","last_became_active":"2020-06-03T18:46:26.512256+0000","last_became_peered":"2020-06-03T18:46:26.512256+0000","last_unstale":"2020-06-03T18:46:26.513333+0000","last_undegraded":"2020-06-03T18:46:26.513333+0000","last_fullsized":"2020-06-03T18:46:26.513333+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.506847+0000","last_change":"2020-06-03T18:46:26.506847+0000","last_active":"2020-06-03T18:46:26.506847+0000","last_peered":"2020-06-03T18:46:26.506847+0000","last_clean":"2020-06-03T18:46:26.506847+0000","last_became_active":"2020-06-03T18:46:26.506433+0000","last_became_peered":"2020-06-03T18:46:26.506433+0000","last_unstale":"2020-06-03T18:46:26.506847+0000","last_undegraded":"2020-06-03T18:46:26.506847+0000","last_fullsized":"2020-06-03T18:46:26.506847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.505214+0000","last_change":"2020-06-03T18:46:23.377239+0000","last_active":"2020-06-03T18:46:26.505214+0000","last_peered":"2020-06-03T18:46:26.505214+0000","last_clean":"2020-06-03T18:46:26.505214+0000","last_became_active":"2020-06-03T18:46:23.375808+0000","last_became_peered":"2020-06-03T18:46:23.375808+0000","last_unstale":"2020-06-03T18:46:26.505214+0000","last_undegraded":"2020-06-03T18:46:26.505214+0000","last_fullsized":"2020-06-03T18:46:26.505214+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":6,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:19.438815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:19.438815+0000","last_clean_scrub_stamp":"2020-06-03T18:46:19.438815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.510673+0000","last_change":"2020-06-03T18:46:26.510289+0000","last_active":"2020-06-03T18:46:26.510673+0000","last_peered":"2020-06-03T18:46:26.510673+0000","last_clean":"2020-06-03T18:46:26.510673+0000","last_became_active":"2020-06-03T18:46:26.509885+0000","last_became_peered":"2020-06-03T18:46:26.509885+0000","last_unstale":"2020-06-03T18:46:26.510673+0000","last_undegraded":"2020-06-03T18:46:26.510673+0000","last_fullsized":"2020-06-03T18:46:26.510673+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.506084+0000","last_change":"2020-06-03T18:46:26.506084+0000","last_active":"2020-06-03T18:46:26.506084+0000","last_peered":"2020-06-03T18:46:26.506084+0000","last_clean":"2020-06-03T18:46:26.506084+0000","last_became_active":"2020-06-03T18:46:26.505722+0000","last_became_peered":"2020-06-03T18:46:26.505722+0000","last_unstale":"2020-06-03T18:46:26.506084+0000","last_undegraded":"2020-06-03T18:46:26.506084+0000","last_fullsized":"2020-06-03T18:46:26.506084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.510845+0000","last_change":"2020-06-03T18:46:26.510845+0000","last_active":"2020-06-03T18:46:26.510845+0000","last_peered":"2020-06-03T18:46:26.510845+0000","last_clean":"2020-06-03T18:46:26.510845+0000","last_became_active":"2020-06-03T18:46:26.510493+0000","last_became_peered":"2020-06-03T18:46:26.510493+0000","last_unstale":"2020-06-03T18:46:26.510845+0000","last_undegraded":"2020-06-03T18:46:26.510845+0000","last_fullsized":"2020-06-03T18:46:26.510845+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.509294+0000","last_change":"2020-06-03T18:46:26.508654+0000","last_active":"2020-06-03T18:46:26.509294+0000","last_peered":"2020-06-03T18:46:26.509294+0000","last_clean":"2020-06-03T18:46:26.509294+0000","last_became_active":"2020-06-03T18:46:26.508274+0000","last_became_peered":"2020-06-03T18:46:26.508274+0000","last_unstale":"2020-06-03T18:46:26.509294+0000","last_undegraded":"2020-06-03T18:46:26.509294+0000","last_fullsized":"2020-06-03T18:46:26.509294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]}]}]}} 2020-06-03T18:46:31.091 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-06-03T18:46:31.344 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:31.346 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2020-06-03T18:46:31.356 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-06-03T18:46:31.099190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":93582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.592846"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513413+0000","last_change":"2020-06-03T18:46:26.513069+0000","last_active":"2020-06-03T18:46:26.513413+0000","last_peered":"2020-06-03T18:46:26.513413+0000","last_clean":"2020-06-03T18:46:26.513413+0000","last_became_active":"2020-06-03T18:46:26.512547+0000","last_became_peered":"2020-06-03T18:46:26.512547+0000","last_unstale":"2020-06-03T18:46:26.513413+0000","last_undegraded":"2020-06-03T18:46:26.513413+0000","last_fullsized":"2020-06-03T18:46:26.513413+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513294+0000","last_change":"2020-06-03T18:46:26.512733+0000","last_active":"2020-06-03T18:46:26.513294+0000","last_peered":"2020-06-03T18:46:26.513294+0000","last_clean":"2020-06-03T18:46:26.513294+0000","last_became_active":"2020-06-03T18:46:26.511498+0000","last_became_peered":"2020-06-03T18:46:26.511498+0000","last_unstale":"2020-06-03T18:46:26.513294+0000","last_undegraded":"2020-06-03T18:46:26.513294+0000","last_fullsized":"2020-06-03T18:46:26.513294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.513333+0000","last_change":"2020-06-03T18:46:26.512686+0000","last_active":"2020-06-03T18:46:26.513333+0000","last_peered":"2020-06-03T18:46:26.513333+0000","last_clean":"2020-06-03T18:46:26.513333+0000","last_became_active":"2020-06-03T18:46:26.512256+0000","last_became_peered":"2020-06-03T18:46:26.512256+0000","last_unstale":"2020-06-03T18:46:26.513333+0000","last_undegraded":"2020-06-03T18:46:26.513333+0000","last_fullsized":"2020-06-03T18:46:26.513333+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.506847+0000","last_change":"2020-06-03T18:46:26.506847+0000","last_active":"2020-06-03T18:46:26.506847+0000","last_peered":"2020-06-03T18:46:26.506847+0000","last_clean":"2020-06-03T18:46:26.506847+0000","last_became_active":"2020-06-03T18:46:26.506433+0000","last_became_peered":"2020-06-03T18:46:26.506433+0000","last_unstale":"2020-06-03T18:46:26.506847+0000","last_undegraded":"2020-06-03T18:46:26.506847+0000","last_fullsized":"2020-06-03T18:46:26.506847+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.505214+0000","last_change":"2020-06-03T18:46:23.377239+0000","last_active":"2020-06-03T18:46:26.505214+0000","last_peered":"2020-06-03T18:46:26.505214+0000","last_clean":"2020-06-03T18:46:26.505214+0000","last_became_active":"2020-06-03T18:46:23.375808+0000","last_became_peered":"2020-06-03T18:46:23.375808+0000","last_unstale":"2020-06-03T18:46:26.505214+0000","last_undegraded":"2020-06-03T18:46:26.505214+0000","last_fullsized":"2020-06-03T18:46:26.505214+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":6,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:19.438815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:19.438815+0000","last_clean_scrub_stamp":"2020-06-03T18:46:19.438815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.510673+0000","last_change":"2020-06-03T18:46:26.510289+0000","last_active":"2020-06-03T18:46:26.510673+0000","last_peered":"2020-06-03T18:46:26.510673+0000","last_clean":"2020-06-03T18:46:26.510673+0000","last_became_active":"2020-06-03T18:46:26.509885+0000","last_became_peered":"2020-06-03T18:46:26.509885+0000","last_unstale":"2020-06-03T18:46:26.510673+0000","last_undegraded":"2020-06-03T18:46:26.510673+0000","last_fullsized":"2020-06-03T18:46:26.510673+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.506084+0000","last_change":"2020-06-03T18:46:26.506084+0000","last_active":"2020-06-03T18:46:26.506084+0000","last_peered":"2020-06-03T18:46:26.506084+0000","last_clean":"2020-06-03T18:46:26.506084+0000","last_became_active":"2020-06-03T18:46:26.505722+0000","last_became_peered":"2020-06-03T18:46:26.505722+0000","last_unstale":"2020-06-03T18:46:26.506084+0000","last_undegraded":"2020-06-03T18:46:26.506084+0000","last_fullsized":"2020-06-03T18:46:26.506084+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.510845+0000","last_change":"2020-06-03T18:46:26.510845+0000","last_active":"2020-06-03T18:46:26.510845+0000","last_peered":"2020-06-03T18:46:26.510845+0000","last_clean":"2020-06-03T18:46:26.510845+0000","last_became_active":"2020-06-03T18:46:26.510493+0000","last_became_peered":"2020-06-03T18:46:26.510493+0000","last_unstale":"2020-06-03T18:46:26.510845+0000","last_undegraded":"2020-06-03T18:46:26.510845+0000","last_fullsized":"2020-06-03T18:46:26.510845+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:26.509294+0000","last_change":"2020-06-03T18:46:26.508654+0000","last_active":"2020-06-03T18:46:26.509294+0000","last_peered":"2020-06-03T18:46:26.509294+0000","last_clean":"2020-06-03T18:46:26.509294+0000","last_became_active":"2020-06-03T18:46:26.508274+0000","last_became_peered":"2020-06-03T18:46:26.508274+0000","last_unstale":"2020-06-03T18:46:26.509294+0000","last_undegraded":"2020-06-03T18:46:26.509294+0000","last_fullsized":"2020-06-03T18:46:26.509294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:25.484164+0000","last_clean_scrub_stamp":"2020-06-03T18:46:25.484164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":30023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.887}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2170000000000001}]}]}]}} 2020-06-03T18:46:31.357 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-06-03T18:46:31.358 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-06-03T18:46:31.358 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-06-03T18:46:31.358 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-06-03T18:46:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:31.676 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-06-03T18:46:31.677 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-06-03T18:46:31.677 INFO:teuthology.run_tasks:Running task ceph... 2020-06-03T18:46:31.689 INFO:tasks.ceph:Creating ceph cluster c2... 2020-06-03T18:46:31.690 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-06-03T18:46:31.690 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi/5116016', 'branch': 'master', 'description': 'rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'email': None, 'first_in_suite': False, 'job_id': '5116016', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-06-03_18:30:30-rgw:multisite-master-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'master'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'fix_issue_43536', 'sha1': '390027420862af043e1952c99341e4057b1eb4e0'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'fix_issue_43536', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_fix_issue_43536/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '390027420862af043e1952c99341e4057b1eb4e0', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '230b0bb1973ab49f8bd55921d5cad7dec0204c35', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.8436', 'targets': {'smithi112.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/nBEuRLx53Ty/6d9ONoTJd876aJq75fLqp/5Y+tmafsmNhROmBI6pG9rUzCKDGOM2iueSI4SFXe4kV2VTqfftCY4vrWRdK4Xy0gZVyh+6pRaH5SbIDei5TAaN1M16VHvn/QJ4dYBxl1Wh4PpmlSFZVA37UVHaOHmHdVVUNdxteqrihmljgDL6K5FBsxncLzUzmVvnDcVKvcJnB0aq3n8DCzvswlf2fwQm6lJ7VO5XPxsa4n3PZeh/YIcAyyNZyADQUeBHrOfl2Zv2hFoQuODijEcyUTSMYkwvuZZ+2ILyGbpfmK1m6z91t62jgnU1PAg2OAtnRhbdM6ZHb+45DEKr', 'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiCUn2qnQjdIV5jntilANWO2UBbEruuFZKvHh00cy0SBueJd7dfgIIHf+B/h5JIO0ikBsLwdmTbIbbiZfXZx1lJvxHZWqe9yTUxej/ILchRWjVs30LngjD7lOwmoHrucD49q4sOcg8mDNLvyR+IQF8WvmGqeGEurxE34fX0KaEPlQgT1ivivdM3iXC71lELrZ+66UdcMUI766Buh3ZsiI3b7BwiaU14boJqHU4L67I5S0s+ibfGMTXAbqoCjfSHvBrcE+t2tx9D6In+E9C6cp16f1Avgi8uoZY5frwkY7D24EvOFIF4Xwu/VAcr30m83iiYgvjr9Y6Muii4pljWRsr'}} 2020-06-03T18:46:31.691 INFO:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-06-03T18:46:31.694 INFO:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-06-03T18:46:31.701 INFO:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2020-06-03T18:46:31.743 INFO:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2020-06-03T18:46:31.806 DEBUG:teuthology.orchestra.remote:smithi061:/scratch_devs is 72B 2020-06-03T18:46:31.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-03T18:46:31.824 INFO:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-03T18:46:31.832 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-03T18:46:31.832 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:31.832 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 345 Links: 1 2020-06-03T18:46:31.832 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:31.833 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2020-06-03 18:43:40.369307907 +0000 2020-06-03T18:46:31.833 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2020-06-03 18:43:40.245245908 +0000 2020-06-03T18:46:31.833 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2020-06-03 18:43:40.245245908 +0000 2020-06-03T18:46:31.833 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2020-06-03T18:46:31.842 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2020-06-03T18:46:31.842 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2020-06-03T18:46:31.842 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000330311 s, 1.6 MB/s 2020-06-03T18:46:31.846 INFO:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-03T18:46:31.897 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-03T18:46:31.897 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:31.898 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 352 Links: 1 2020-06-03T18:46:31.898 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:31.898 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2020-06-03 18:43:40.633439904 +0000 2020-06-03T18:46:31.899 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2020-06-03 18:43:40.509377905 +0000 2020-06-03T18:46:31.899 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2020-06-03 18:43:40.509377905 +0000 2020-06-03T18:46:31.899 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2020-06-03T18:46:31.906 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2020-06-03T18:46:31.907 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2020-06-03T18:46:31.907 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000361581 s, 1.4 MB/s 2020-06-03T18:46:31.912 INFO:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-03T18:46:31.960 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-03T18:46:31.961 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:31.961 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 358 Links: 1 2020-06-03T18:46:31.961 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:31.961 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2020-06-03 18:43:40.881563901 +0000 2020-06-03T18:46:31.962 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2020-06-03 18:43:40.761503903 +0000 2020-06-03T18:46:31.962 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2020-06-03 18:43:40.761503903 +0000 2020-06-03T18:46:31.962 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2020-06-03T18:46:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2020-06-03T18:46:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2020-06-03T18:46:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000359274 s, 1.4 MB/s 2020-06-03T18:46:31.975 INFO:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-03T18:46:32.024 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-03T18:46:32.025 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-03T18:46:32.025 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 364 Links: 1 2020-06-03T18:46:32.025 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-03T18:46:32.025 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2020-06-03 18:43:41.125685899 +0000 2020-06-03T18:46:32.026 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2020-06-03 18:43:41.021633899 +0000 2020-06-03T18:46:32.026 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2020-06-03 18:43:41.021633899 +0000 2020-06-03T18:46:32.026 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2020-06-03T18:46:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2020-06-03T18:46:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2020-06-03T18:46:32.035 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000330096 s, 1.6 MB/s 2020-06-03T18:46:32.039 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:32.039 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-03T18:46:32.040 INFO:tasks.ceph:Generating config... 2020-06-03T18:46:32.041 INFO:tasks.ceph:[client] debug rgw = 20 2020-06-03T18:46:32.041 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-06-03T18:46:32.042 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-06-03T18:46:32.042 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:32.043 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-06-03T18:46:32.043 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-06-03T18:46:32.043 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-06-03T18:46:32.044 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-06-03T18:46:32.044 INFO:tasks.ceph:[client] setgroup = ceph 2020-06-03T18:46:32.045 INFO:tasks.ceph:[client] setuser = ceph 2020-06-03T18:46:32.045 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-06-03T18:46:32.045 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-06-03T18:46:32.046 INFO:tasks.ceph:[mgr] debug ms = 1 2020-06-03T18:46:32.046 INFO:tasks.ceph:[mon] debug mon = 20 2020-06-03T18:46:32.047 INFO:tasks.ceph:[mon] debug ms = 1 2020-06-03T18:46:32.047 INFO:tasks.ceph:[mon] debug paxos = 20 2020-06-03T18:46:32.048 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-06-03T18:46:32.048 INFO:tasks.ceph:[osd] debug filestore = 20 2020-06-03T18:46:32.048 INFO:tasks.ceph:[osd] debug journal = 20 2020-06-03T18:46:32.049 INFO:tasks.ceph:[osd] debug ms = 20 2020-06-03T18:46:32.049 INFO:tasks.ceph:[osd] debug osd = 25 2020-06-03T18:46:32.050 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-06-03T18:46:32.050 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-06-03T18:46:32.050 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-06-03T18:46:32.051 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:32.051 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:32.051 INFO:tasks.ceph:Setting up c2.mon.a... 2020-06-03T18:46:32.052 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-06-03T18:46:32.114 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c2.keyring 2020-06-03T18:46:32.116 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-06-03T18:46:32.203 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-03T18:46:32.265 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.61')] 2020-06-03T18:46:32.266 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.61 --print /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:32.333 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:32.333 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: generated fsid e0db1e0b-17e4-446f-a403-483ec9068b06 2020-06-03T18:46:32.334 INFO:teuthology.orchestra.run.smithi061.stdout:epoch 0 2020-06-03T18:46:32.334 INFO:teuthology.orchestra.run.smithi061.stdout:fsid e0db1e0b-17e4-446f-a403-483ec9068b06 2020-06-03T18:46:32.335 INFO:teuthology.orchestra.run.smithi061.stdout:last_changed 2020-06-03T18:46:32.336970+0000 2020-06-03T18:46:32.335 INFO:teuthology.orchestra.run.smithi061.stdout:created 2020-06-03T18:46:32.336970+0000 2020-06-03T18:46:32.335 INFO:teuthology.orchestra.run.smithi061.stdout:min_mon_release 0 (unknown) 2020-06-03T18:46:32.336 INFO:teuthology.orchestra.run.smithi061.stdout:0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.a 2020-06-03T18:46:32.336 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-06-03T18:46:32.337 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID e0db1e0b-17e4-446f-a403-483ec9068b06... 2020-06-03T18:46:32.338 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-06-03T18:46:32.380 INFO:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-06-03T18:46:32.411 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2020-06-03T18:46:32.411 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2020-06-03T18:46:32.411 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-03T18:46:32.411 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.412 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2020-06-03T18:46:32.413 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2020-06-03T18:46:32.413 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.413 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2020-06-03T18:46:32.413 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2020-06-03T18:46:32.414 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.414 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-03T18:46:32.414 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2020-06-03T18:46:32.414 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.414 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2020-06-03T18:46:32.415 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-03T18:46:32.415 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.415 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2020-06-03T18:46:32.415 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.415 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2020-06-03T18:46:32.416 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2020-06-03T18:46:32.417 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-03T18:46:32.417 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-03T18:46:32.417 INFO:teuthology.orchestra.run.smithi061.stdout: mon_allow_pool_size_one = true 2020-06-03T18:46:32.417 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.417 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-03T18:46:32.418 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.418 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2020-06-03T18:46:32.418 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.418 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2020-06-03T18:46:32.418 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = 172.21.15.61 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: osd heartbeat grace = 40 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = e0db1e0b-17e4-446f-a403-483ec9068b06 2020-06-03T18:46:32.419 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.420 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2020-06-03T18:46:32.420 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2020-06-03T18:46:32.420 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.420 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2020-06-03T18:46:32.420 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2020-06-03T18:46:32.421 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2020-06-03T18:46:32.430 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2020-06-03T18:46:32.431 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.431 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2020-06-03T18:46:32.431 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2020-06-03T18:46:32.431 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.431 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.432 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2020-06-03T18:46:32.433 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2020-06-03T18:46:32.433 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.433 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2020-06-03T18:46:32.433 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2020-06-03T18:46:32.433 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 20 2020-06-03T18:46:32.434 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 20 2020-06-03T18:46:32.434 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 20 2020-06-03T18:46:32.434 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 25 2020-06-03T18:46:32.434 INFO:teuthology.orchestra.run.smithi061.stdout: osd fast shutdown = False 2020-06-03T18:46:32.434 INFO:teuthology.orchestra.run.smithi061.stdout: rocksdb delete range threshold = 0 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2020-06-03T18:46:32.435 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2020-06-03T18:46:32.436 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:32.436 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:32.436 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2020-06-03T18:46:32.436 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.436 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2020-06-03T18:46:32.437 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2020-06-03T18:46:32.438 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:32.438 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2020-06-03T18:46:32.438 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2020-06-03T18:46:32.438 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:32.438 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd crush smoke test = False 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2020-06-03T18:46:32.439 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-03T18:46:32.440 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-03T18:46:32.440 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw = 20 2020-06-03T18:46:32.440 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt require ssl = False 2020-06-03T18:46:32.440 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms backend = testing 2020-06-03T18:46:32.441 INFO:teuthology.orchestra.run.smithi061.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:32.441 INFO:teuthology.orchestra.run.smithi061.stdout: rgw curl low speed time = 300 2020-06-03T18:46:32.441 INFO:teuthology.orchestra.run.smithi061.stdout: rgw data log num shards = 4 2020-06-03T18:46:32.442 INFO:teuthology.orchestra.run.smithi061.stdout: rgw md log max shards = 4 2020-06-03T18:46:32.442 INFO:teuthology.orchestra.run.smithi061.stdout: rgw sync log trim interval = 0 2020-06-03T18:46:32.442 INFO:teuthology.orchestra.run.smithi061.stdout: setgroup = ceph 2020-06-03T18:46:32.442 INFO:teuthology.orchestra.run.smithi061.stdout: setuser = ceph 2020-06-03T18:46:32.443 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2020-06-03T18:46:32.443 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2020-06-03T18:46:32.443 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10 2020-06-03T18:46:32.444 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2020-06-03T18:46:32.444 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:32.444 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2020-06-03T18:46:32.444 INFO:teuthology.orchestra.run.smithi061.stdout: osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:32.447 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2020-06-03T18:46:32.447 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2020-06-03T18:46:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-03T18:46:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2020-06-03T18:46:32.448 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2020-06-03T18:46:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.449 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2020-06-03T18:46:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2020-06-03T18:46:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2020-06-03T18:46:32.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2020-06-03T18:46:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2020-06-03T18:46:32.451 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-03T18:46:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2020-06-03T18:46:32.452 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2020-06-03T18:46:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-03T18:46:32.453 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2020-06-03T18:46:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.454 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2020-06-03T18:46:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2020-06-03T18:46:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2020-06-03T18:46:32.455 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2020-06-03T18:46:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2020-06-03T18:46:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2020-06-03T18:46:32.456 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2020-06-03T18:46:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-03T18:46:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-03T18:46:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2020-06-03T18:46:32.457 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-03T18:46:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.458 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2020-06-03T18:46:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2020-06-03T18:46:32.459 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2020-06-03T18:46:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2020-06-03T18:46:32.460 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.61 2020-06-03T18:46:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: osd heartbeat grace = 40 2020-06-03T18:46:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = e0db1e0b-17e4-446f-a403-483ec9068b06 2020-06-03T18:46:32.461 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.462 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2020-06-03T18:46:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2020-06-03T18:46:32.462 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2020-06-03T18:46:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2020-06-03T18:46:32.463 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2020-06-03T18:46:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2020-06-03T18:46:32.464 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2020-06-03T18:46:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2020-06-03T18:46:32.465 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2020-06-03T18:46:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2020-06-03T18:46:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2020-06-03T18:46:32.466 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2020-06-03T18:46:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2020-06-03T18:46:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.467 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2020-06-03T18:46:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2020-06-03T18:46:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2020-06-03T18:46:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.468 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2020-06-03T18:46:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2020-06-03T18:46:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: debug filestore = 20 2020-06-03T18:46:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: debug journal = 20 2020-06-03T18:46:32.469 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 20 2020-06-03T18:46:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 25 2020-06-03T18:46:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: osd fast shutdown = False 2020-06-03T18:46:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2020-06-03T18:46:32.470 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.470 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:32.471 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2020-06-03T18:46:32.472 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2020-06-03T18:46:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2020-06-03T18:46:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2020-06-03T18:46:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2020-06-03T18:46:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2020-06-03T18:46:32.473 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2020-06-03T18:46:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2020-06-03T18:46:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2020-06-03T18:46:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd crush smoke test = False 2020-06-03T18:46:32.474 INFO:teuthology.orchestra.run.smithi112.stdout: 2020-06-03T18:46:32.474 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2020-06-03T18:46:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2020-06-03T18:46:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2020-06-03T18:46:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2020-06-03T18:46:32.475 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-03T18:46:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-03T18:46:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2020-06-03T18:46:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2020-06-03T18:46:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2020-06-03T18:46:32.476 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-03T18:46:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2020-06-03T18:46:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2020-06-03T18:46:32.477 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2020-06-03T18:46:32.478 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2020-06-03T18:46:32.478 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2020-06-03T18:46:32.478 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2020-06-03T18:46:32.478 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2020-06-03T18:46:32.479 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2020-06-03T18:46:32.479 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2020-06-03T18:46:32.479 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2020-06-03T18:46:32.479 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2020-06-03T18:46:32.479 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2020-06-03T18:46:32.480 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2020-06-03T18:46:32.480 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-06-03T18:46:32.481 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-06-03T18:46:32.539 INFO:tasks.ceph:Copying monmap to all nodes... 2020-06-03T18:46:32.548 DEBUG:teuthology.orchestra.remote:smithi061:/etc/ceph/c2.keyring is 206B 2020-06-03T18:46:32.566 DEBUG:teuthology.orchestra.remote:smithi061:/home/ubuntu/cephtest/c2.monmap is 179B 2020-06-03T18:46:32.576 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2020-06-03T18:46:32.576 INFO:teuthology.orchestra.run.smithi112:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-03T18:46:32.631 INFO:teuthology.orchestra.run.smithi112:> cat > /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:32.721 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:46:32.721 INFO:teuthology.orchestra.run.smithi061:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-03T18:46:32.748 INFO:teuthology.orchestra.run.smithi061:> cat > /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:32.835 INFO:tasks.ceph:Setting up mon nodes... 2020-06-03T18:46:32.836 INFO:tasks.ceph:Setting up mgr nodes... 2020-06-03T18:46:32.836 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-06-03T18:46:32.923 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-06-03T18:46:32.926 INFO:tasks.ceph:Setting up mds nodes... 2020-06-03T18:46:32.926 INFO:tasks.ceph_client:Setting up client nodes... 2020-06-03T18:46:32.927 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-06-03T18:46:32.957 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c2.client.0.keyring 2020-06-03T18:46:32.970 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-06-03T18:46:33.051 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/c2.client.1.keyring 2020-06-03T18:46:33.063 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-06-03T18:46:33.063 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-03T18:46:33.064 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-06-03T18:46:33.127 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:33.127 INFO:tasks.ceph:role: c2.osd.0 2020-06-03T18:46:33.128 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:46:33.128 INFO:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-03T18:46:33.653 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:33.900 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:33.900 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:33.900 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:33.901 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:33.901 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:33.901 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:33.901 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:33.901 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:33.902 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2020-06-03T18:46:33.903 INFO:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-06-03T18:46:33.985 INFO:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-06-03T18:46:33.998 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:33.999 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-06-03T18:46:34.057 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:34.057 INFO:tasks.ceph:role: c2.osd.1 2020-06-03T18:46:34.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:46:34.058 INFO:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-03T18:46:34.560 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:34.560 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:34.561 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:34.561 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:34.561 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:34.562 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:34.562 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:34.562 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:34.563 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:34.564 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2020-06-03T18:46:34.565 INFO:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-06-03T18:46:34.616 INFO:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-06-03T18:46:34.669 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:34.671 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-06-03T18:46:34.729 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-03T18:46:34.730 INFO:tasks.ceph:role: c2.osd.2 2020-06-03T18:46:34.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com 2020-06-03T18:46:34.731 INFO:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-03T18:46:35.271 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-03T18:46:35.271 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-03T18:46:35.271 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-03T18:46:35.272 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-03T18:46:35.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2020-06-03T18:46:35.274 INFO:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-06-03T18:46:35.316 INFO:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-06-03T18:46:35.371 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2020-06-03T18:46:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-03T18:46:35.373 INFO:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:35.482 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.478+0000 7fdd26474d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-06-03T18:46:35.483 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.482+0000 7fdd26474d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-06-03T18:46:35.483 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.482+0000 7fdd26474d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-03T18:46:35.484 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.482+0000 7fdd26474d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-03T18:46:35.484 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.482+0000 7fdd26474d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-03T18:46:35.491 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:35.490+0000 7fdd26474d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-06-03T18:46:37.426 INFO:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-06-03T18:46:37.446 INFO:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:37.528 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-06-03T18:46:37.529 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-06-03T18:46:37.529 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-03T18:46:37.529 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-03T18:46:37.530 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-03T18:46:37.531 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:37.526+0000 7f43caa0cd80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-06-03T18:46:39.445 INFO:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-06-03T18:46:39.505 INFO:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-06-03T18:46:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-06-03T18:46:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-03T18:46:39.585 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-03T18:46:39.585 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-03T18:46:39.585 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:46:39.582+0000 7f08081bbd80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-06-03T18:46:41.505 INFO:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-06-03T18:46:41.561 INFO:tasks.ceph:Reading keys from all nodes... 2020-06-03T18:46:41.562 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:41.607 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.3AHoWyxNnL 2020-06-03T18:46:41.607 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.3AHoWyxNnL 2020-06-03T18:46:41.665 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.3AHoWyxNnL 2020-06-03T18:46:41.769 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.3AHoWyxNnL is 56B 2020-06-03T18:46:41.783 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.3AHoWyxNnL 2020-06-03T18:46:41.790 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:41.848 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.on7TDdgbDQ 2020-06-03T18:46:41.848 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.on7TDdgbDQ 2020-06-03T18:46:41.905 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.on7TDdgbDQ 2020-06-03T18:46:42.013 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.on7TDdgbDQ is 56B 2020-06-03T18:46:42.024 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.on7TDdgbDQ 2020-06-03T18:46:42.031 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:42.079 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.AcbFtOD7T2 2020-06-03T18:46:42.079 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.AcbFtOD7T2 2020-06-03T18:46:42.136 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.AcbFtOD7T2 2020-06-03T18:46:42.241 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.AcbFtOD7T2 is 56B 2020-06-03T18:46:42.252 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.AcbFtOD7T2 2020-06-03T18:46:42.258 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:42.307 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.aJ8w9DOGjx 2020-06-03T18:46:42.308 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.aJ8w9DOGjx 2020-06-03T18:46:42.369 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.aJ8w9DOGjx 2020-06-03T18:46:42.474 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.aJ8w9DOGjx is 56B 2020-06-03T18:46:42.486 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.aJ8w9DOGjx 2020-06-03T18:46:42.542 DEBUG:teuthology.orchestra.remote:smithi061:/etc/ceph/c2.client.0.keyring is 59B 2020-06-03T18:46:42.603 DEBUG:teuthology.orchestra.remote:smithi061:/etc/ceph/c2.client.1.keyring is 59B 2020-06-03T18:46:42.614 INFO:tasks.ceph:Adding keys to all mons... 2020-06-03T18:46:42.614 INFO:teuthology.orchestra.run.smithi061:> sudo tee -a /etc/ceph/c2.keyring 2020-06-03T18:46:42.666 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.x] 2020-06-03T18:46:42.667 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAI8NdeZGlKNxAAfltDnGqb4BqMf8CNmJVgkQ== 2020-06-03T18:46:42.667 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2020-06-03T18:46:42.667 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAL8NdeoVr2HBAABfO/Ox7MKCbgUeN2IhjnXA== 2020-06-03T18:46:42.668 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2020-06-03T18:46:42.668 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAN8NdeLCCyHxAAQFyPlRxMcDHllfb5bYE+OA== 2020-06-03T18:46:42.668 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2020-06-03T18:46:42.668 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAP8Nde1CoAIxAA4man9dbeGMnaoRg5FDN/Iw== 2020-06-03T18:46:42.669 INFO:teuthology.orchestra.run.smithi061.stdout:[client.0] 2020-06-03T18:46:42.669 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAI8NdeeqNIORAAoVVLyIq4LYNW4ETbXuf1DA== 2020-06-03T18:46:42.669 INFO:teuthology.orchestra.run.smithi061.stdout:[client.1] 2020-06-03T18:46:42.669 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQAJ8NdeZbxHAxAA+kx4xYzN+haSf5VIOeG2nQ== 2020-06-03T18:46:42.671 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-03T18:46:42.756 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:42.840 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:42.926 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-03T18:46:43.012 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-03T18:46:43.099 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-03T18:46:43.141 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-06-03T18:46:43.142 INFO:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-06-03T18:46:43.199 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-06-03T18:46:43.303 INFO:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-06-03T18:46:43.316 INFO:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-06-03T18:46:43.368 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-06-03T18:46:43.368 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-06-03T18:46:43.369 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-06-03T18:46:43.411 INFO:tasks.ceph.c2.mon.a:Started 2020-06-03T18:46:43.412 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-06-03T18:46:43.413 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-06-03T18:46:43.413 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-06-03T18:46:43.417 INFO:tasks.ceph.c2.mgr.x:Started 2020-06-03T18:46:43.418 INFO:tasks.ceph:Setting crush tunables to default 2020-06-03T18:46:43.418 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd crush tunables default 2020-06-03T18:46:43.756 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2020-06-03T18:46:43.771 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-06-03T18:46:43.771 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:43.777 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.3AbTlh13ud 2020-06-03T18:46:43.777 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.3AbTlh13ud 2020-06-03T18:46:43.829 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.3AbTlh13ud 2020-06-03T18:46:43.939 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.3AbTlh13ud is 37B 2020-06-03T18:46:43.959 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.3AbTlh13ud 2020-06-03T18:46:43.966 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:44.015 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.3Nj7Gvs0Wn 2020-06-03T18:46:44.016 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.3Nj7Gvs0Wn 2020-06-03T18:46:44.072 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.3Nj7Gvs0Wn 2020-06-03T18:46:44.176 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.3Nj7Gvs0Wn is 37B 2020-06-03T18:46:44.187 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.3Nj7Gvs0Wn 2020-06-03T18:46:44.194 INFO:teuthology.orchestra.run.smithi061:> mktemp 2020-06-03T18:46:44.244 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.falIr2rYOd 2020-06-03T18:46:44.244 INFO:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.falIr2rYOd 2020-06-03T18:46:44.300 INFO:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.falIr2rYOd 2020-06-03T18:46:44.406 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.falIr2rYOd is 37B 2020-06-03T18:46:44.418 INFO:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.falIr2rYOd 2020-06-03T18:46:44.425 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd new 88a68801-2b92-4677-9f6c-821a8bf706ff 0 2020-06-03T18:46:44.674 INFO:teuthology.orchestra.run.smithi061.stdout:0 2020-06-03T18:46:44.686 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd new 1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7 1 2020-06-03T18:46:44.897 INFO:teuthology.orchestra.run.smithi061.stdout:1 2020-06-03T18:46:44.909 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd new 586de0dd-68f1-44b0-b3fe-1746ee7d490e 2 2020-06-03T18:46:45.113 INFO:teuthology.orchestra.run.smithi061.stdout:2 2020-06-03T18:46:45.127 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-06-03T18:46:45.128 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-06-03T18:46:45.130 INFO:tasks.ceph.c2.osd.0:Started 2020-06-03T18:46:45.130 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-06-03T18:46:45.131 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-06-03T18:46:45.133 INFO:tasks.ceph.c2.osd.1:Started 2020-06-03T18:46:45.134 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-06-03T18:46:45.134 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-06-03T18:46:45.136 INFO:tasks.ceph.c2.osd.2:Started 2020-06-03T18:46:45.137 INFO:tasks.ceph:Waiting for OSDs to come up 2020-06-03T18:46:45.440 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-06-03T18:46:45.622 INFO:teuthology.misc.health.smithi061.stdout: 2020-06-03T18:46:45.623 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":5,"fsid":"e0db1e0b-17e4-446f-a403-483ec9068b06","created":"2020-06-03T18:46:43.548068+0000","modified":"2020-06-03T18:46:45.114541+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"88a68801-2b92-4677-9f6c-821a8bf706ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"586de0dd-68f1-44b0-b3fe-1746ee7d490e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:45.627 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-06-03T18:46:45.728 INFO:tasks.ceph.c2.osd.1.smithi061.stderr:2020-06-03T18:46:45.726+0000 7f35dfef8d80 -1 Falling back to public interface 2020-06-03T18:46:45.759 INFO:tasks.ceph.c2.osd.2.smithi061.stderr:2020-06-03T18:46:45.758+0000 7f576db64d80 -1 Falling back to public interface 2020-06-03T18:46:45.760 INFO:tasks.ceph.c2.osd.0.smithi061.stderr:2020-06-03T18:46:45.758+0000 7fb62d908d80 -1 Falling back to public interface 2020-06-03T18:46:46.349 INFO:tasks.ceph.c2.osd.1.smithi061.stderr:2020-06-03T18:46:46.346+0000 7f35dfef8d80 -1 osd.1 0 log_to_monitors {default=true} 2020-06-03T18:46:46.402 INFO:tasks.ceph.c2.osd.0.smithi061.stderr:2020-06-03T18:46:46.402+0000 7fb62d908d80 -1 osd.0 0 log_to_monitors {default=true} 2020-06-03T18:46:46.405 INFO:tasks.ceph.c2.osd.2.smithi061.stderr:2020-06-03T18:46:46.402+0000 7f576db64d80 -1 osd.2 0 log_to_monitors {default=true} 2020-06-03T18:46:47.575 INFO:tasks.ceph.c2.osd.1.smithi061.stderr:2020-06-03T18:46:47.574+0000 7f35d73c3700 -1 osd.1 0 waiting for initial osdmap 2020-06-03T18:46:47.575 INFO:tasks.ceph.c2.osd.2.smithi061.stderr:2020-06-03T18:46:47.574+0000 7f576502f700 -1 osd.2 0 waiting for initial osdmap 2020-06-03T18:46:47.575 INFO:tasks.ceph.c2.osd.0.smithi061.stderr:2020-06-03T18:46:47.574+0000 7fb624dd3700 -1 osd.0 0 waiting for initial osdmap 2020-06-03T18:46:51.931 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-06-03T18:46:52.198 INFO:teuthology.misc.health.smithi061.stdout: 2020-06-03T18:46:52.199 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":10,"fsid":"e0db1e0b-17e4-446f-a403-483ec9068b06","created":"2020-06-03T18:46:43.548068+0000","modified":"2020-06-03T18:46:50.592320+0000","last_up_change":"2020-06-03T18:46:48.571319+0000","last_in_change":"2020-06-03T18:46:48.571319+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:47.585203+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"88a68801-2b92-4677-9f6c-821a8bf706ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":13755},{"type":"v1","addr":"172.21.15.61:6812","nonce":13755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":13755},{"type":"v1","addr":"172.21.15.61:6815","nonce":13755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":13755},{"type":"v1","addr":"172.21.15.61:6823","nonce":13755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6817","nonce":13755},{"type":"v1","addr":"172.21.15.61:6819","nonce":13755}]},"public_addr":"172.21.15.61:6812/13755","cluster_addr":"172.21.15.61:6815/13755","heartbeat_back_addr":"172.21.15.61:6823/13755","heartbeat_front_addr":"172.21.15.61:6819/13755","state":["exists","up"]},{"osd":1,"uuid":"1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":13756},{"type":"v1","addr":"172.21.15.61:6801","nonce":13756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":13756},{"type":"v1","addr":"172.21.15.61:6803","nonce":13756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":13756},{"type":"v1","addr":"172.21.15.61:6807","nonce":13756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":13756},{"type":"v1","addr":"172.21.15.61:6805","nonce":13756}]},"public_addr":"172.21.15.61:6801/13756","cluster_addr":"172.21.15.61:6803/13756","heartbeat_back_addr":"172.21.15.61:6807/13756","heartbeat_front_addr":"172.21.15.61:6805/13756","state":["exists","up"]},{"osd":2,"uuid":"586de0dd-68f1-44b0-b3fe-1746ee7d490e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":13779},{"type":"v1","addr":"172.21.15.61:6809","nonce":13779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":13779},{"type":"v1","addr":"172.21.15.61:6813","nonce":13779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":13779},{"type":"v1","addr":"172.21.15.61:6821","nonce":13779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":13779},{"type":"v1","addr":"172.21.15.61:6818","nonce":13779}]},"public_addr":"172.21.15.61:6809/13779","cluster_addr":"172.21.15.61:6813/13779","heartbeat_back_addr":"172.21.15.61:6821/13779","heartbeat_front_addr":"172.21.15.61:6818/13779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.447645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.321709+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.443718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:52.210 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-06-03T18:46:52.211 INFO:tasks.ceph:Creating RBD pool 2020-06-03T18:46:52.211 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd pool create rbd 8 2020-06-03T18:46:52.673 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'rbd' created 2020-06-03T18:46:52.687 INFO:teuthology.orchestra.run.smithi061:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-06-03T18:46:53.686 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rbd' on pool 'rbd' 2020-06-03T18:46:53.700 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-06-03T18:46:53.701 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-03T18:46:53.702 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-06-03T18:46:53.971 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:53.972 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"fsid":"e0db1e0b-17e4-446f-a403-483ec9068b06","created":"2020-06-03T18:46:43.548068+0000","modified":"2020-06-03T18:46:53.665241+0000","last_up_change":"2020-06-03T18:46:48.571319+0000","last_in_change":"2020-06-03T18:46:48.571319+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:47.585203+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:52.481760+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"88a68801-2b92-4677-9f6c-821a8bf706ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":13755},{"type":"v1","addr":"172.21.15.61:6812","nonce":13755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":13755},{"type":"v1","addr":"172.21.15.61:6815","nonce":13755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":13755},{"type":"v1","addr":"172.21.15.61:6823","nonce":13755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6817","nonce":13755},{"type":"v1","addr":"172.21.15.61:6819","nonce":13755}]},"public_addr":"172.21.15.61:6812/13755","cluster_addr":"172.21.15.61:6815/13755","heartbeat_back_addr":"172.21.15.61:6823/13755","heartbeat_front_addr":"172.21.15.61:6819/13755","state":["exists","up"]},{"osd":1,"uuid":"1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":13756},{"type":"v1","addr":"172.21.15.61:6801","nonce":13756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":13756},{"type":"v1","addr":"172.21.15.61:6803","nonce":13756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":13756},{"type":"v1","addr":"172.21.15.61:6807","nonce":13756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":13756},{"type":"v1","addr":"172.21.15.61:6805","nonce":13756}]},"public_addr":"172.21.15.61:6801/13756","cluster_addr":"172.21.15.61:6803/13756","heartbeat_back_addr":"172.21.15.61:6807/13756","heartbeat_front_addr":"172.21.15.61:6805/13756","state":["exists","up"]},{"osd":2,"uuid":"586de0dd-68f1-44b0-b3fe-1746ee7d490e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":13779},{"type":"v1","addr":"172.21.15.61:6809","nonce":13779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":13779},{"type":"v1","addr":"172.21.15.61:6813","nonce":13779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":13779},{"type":"v1","addr":"172.21.15.61:6821","nonce":13779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":13779},{"type":"v1","addr":"172.21.15.61:6818","nonce":13779}]},"public_addr":"172.21.15.61:6809/13779","cluster_addr":"172.21.15.61:6813/13779","heartbeat_back_addr":"172.21.15.61:6821/13779","heartbeat_front_addr":"172.21.15.61:6818/13779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.447645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.321709+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.443718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:53.984 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-03T18:46:47.585203+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-03T18:46:52.481760+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-06-03T18:46:53.985 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-06-03T18:46:54.257 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2020-06-03T18:46:54.271 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-06-03T18:46:54.543 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2020-06-03T18:46:54.556 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-06-03T18:46:54.556 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-06-03T18:46:54.557 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-06-03T18:46:54.856 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:54.869 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":13554},{"type":"v1","addr":"172.21.15.61:6825","nonce":13554}]},"active_addr":"172.21.15.61:6825/13554","active_change":"2020-06-03T18:46:47.544117+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:latest","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":3298411094}]},{"addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":4057251853}]},{"addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":2730364331}]}]}} 2020-06-03T18:46:54.871 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-06-03T18:46:54.872 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-06-03T18:46:54.873 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-03T18:46:55.137 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:55.138 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":13,"fsid":"e0db1e0b-17e4-446f-a403-483ec9068b06","created":"2020-06-03T18:46:43.548068+0000","modified":"2020-06-03T18:46:54.670972+0000","last_up_change":"2020-06-03T18:46:48.571319+0000","last_in_change":"2020-06-03T18:46:48.571319+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:47.585203+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:52.481760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"88a68801-2b92-4677-9f6c-821a8bf706ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":13755},{"type":"v1","addr":"172.21.15.61:6812","nonce":13755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":13755},{"type":"v1","addr":"172.21.15.61:6815","nonce":13755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":13755},{"type":"v1","addr":"172.21.15.61:6823","nonce":13755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6817","nonce":13755},{"type":"v1","addr":"172.21.15.61:6819","nonce":13755}]},"public_addr":"172.21.15.61:6812/13755","cluster_addr":"172.21.15.61:6815/13755","heartbeat_back_addr":"172.21.15.61:6823/13755","heartbeat_front_addr":"172.21.15.61:6819/13755","state":["exists","up"]},{"osd":1,"uuid":"1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":13756},{"type":"v1","addr":"172.21.15.61:6801","nonce":13756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":13756},{"type":"v1","addr":"172.21.15.61:6803","nonce":13756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":13756},{"type":"v1","addr":"172.21.15.61:6807","nonce":13756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":13756},{"type":"v1","addr":"172.21.15.61:6805","nonce":13756}]},"public_addr":"172.21.15.61:6801/13756","cluster_addr":"172.21.15.61:6803/13756","heartbeat_back_addr":"172.21.15.61:6807/13756","heartbeat_front_addr":"172.21.15.61:6805/13756","state":["exists","up"]},{"osd":2,"uuid":"586de0dd-68f1-44b0-b3fe-1746ee7d490e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":13779},{"type":"v1","addr":"172.21.15.61:6809","nonce":13779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":13779},{"type":"v1","addr":"172.21.15.61:6813","nonce":13779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":13779},{"type":"v1","addr":"172.21.15.61:6821","nonce":13779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":13779},{"type":"v1","addr":"172.21.15.61:6818","nonce":13779}]},"public_addr":"172.21.15.61:6809/13779","cluster_addr":"172.21.15.61:6813/13779","heartbeat_back_addr":"172.21.15.61:6821/13779","heartbeat_front_addr":"172.21.15.61:6818/13779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.447645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.321709+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.443718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:55.151 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-06-03T18:46:55.151 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-03T18:46:55.428 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:55.429 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":13,"fsid":"e0db1e0b-17e4-446f-a403-483ec9068b06","created":"2020-06-03T18:46:43.548068+0000","modified":"2020-06-03T18:46:54.670972+0000","last_up_change":"2020-06-03T18:46:48.571319+0000","last_in_change":"2020-06-03T18:46:48.571319+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-03T18:46:47.585203+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-03T18:46:52.481760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"88a68801-2b92-4677-9f6c-821a8bf706ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":13755},{"type":"v1","addr":"172.21.15.61:6812","nonce":13755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":13755},{"type":"v1","addr":"172.21.15.61:6815","nonce":13755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":13755},{"type":"v1","addr":"172.21.15.61:6823","nonce":13755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6817","nonce":13755},{"type":"v1","addr":"172.21.15.61:6819","nonce":13755}]},"public_addr":"172.21.15.61:6812/13755","cluster_addr":"172.21.15.61:6815/13755","heartbeat_back_addr":"172.21.15.61:6823/13755","heartbeat_front_addr":"172.21.15.61:6819/13755","state":["exists","up"]},{"osd":1,"uuid":"1b6e5ae3-ca71-46f6-bdce-dca3f9b377f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":13756},{"type":"v1","addr":"172.21.15.61:6801","nonce":13756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":13756},{"type":"v1","addr":"172.21.15.61:6803","nonce":13756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":13756},{"type":"v1","addr":"172.21.15.61:6807","nonce":13756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":13756},{"type":"v1","addr":"172.21.15.61:6805","nonce":13756}]},"public_addr":"172.21.15.61:6801/13756","cluster_addr":"172.21.15.61:6803/13756","heartbeat_back_addr":"172.21.15.61:6807/13756","heartbeat_front_addr":"172.21.15.61:6805/13756","state":["exists","up"]},{"osd":2,"uuid":"586de0dd-68f1-44b0-b3fe-1746ee7d490e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":13779},{"type":"v1","addr":"172.21.15.61:6809","nonce":13779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":13779},{"type":"v1","addr":"172.21.15.61:6813","nonce":13779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":13779},{"type":"v1","addr":"172.21.15.61:6821","nonce":13779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":13779},{"type":"v1","addr":"172.21.15.61:6818","nonce":13779}]},"public_addr":"172.21.15.61:6809/13779","cluster_addr":"172.21.15.61:6813/13779","heartbeat_back_addr":"172.21.15.61:6821/13779","heartbeat_front_addr":"172.21.15.61:6818/13779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.447645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.321709+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-03T18:46:47.443718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-03T18:46:55.438 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-06-03T18:46:55.596 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:55.597 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-06-03T18:46:55.763 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:55.763 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-06-03T18:46:55.934 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:55.934 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-03T18:46:56.212 INFO:teuthology.orchestra.run.smithi061.stdout:34359738371 2020-06-03T18:46:56.226 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-06-03T18:46:57.228 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-03T18:46:57.505 INFO:teuthology.orchestra.run.smithi061.stdout:34359738371 2020-06-03T18:46:57.518 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2020-06-03T18:46:58.519 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-03T18:46:58.800 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:58.813 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2020-06-03T18:46:58.814 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-06-03T18:46:59.081 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:59.095 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2020-06-03T18:46:59.096 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-06-03T18:46:59.368 INFO:teuthology.orchestra.run.smithi061.stdout:34359738372 2020-06-03T18:46:59.381 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-06-03T18:46:59.382 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-06-03T18:46:59.382 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-06-03T18:46:59.638 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:59.640 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2020-06-03T18:46:59.657 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-03T18:46:59.555626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":91182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.882374"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.688486+0000","last_change":"2020-06-03T18:46:53.688212+0000","last_active":"2020-06-03T18:46:53.688486+0000","last_peered":"2020-06-03T18:46:53.688486+0000","last_clean":"2020-06-03T18:46:53.688486+0000","last_became_active":"2020-06-03T18:46:53.687776+0000","last_became_peered":"2020-06-03T18:46:53.687776+0000","last_unstale":"2020-06-03T18:46:53.688486+0000","last_undegraded":"2020-06-03T18:46:53.688486+0000","last_fullsized":"2020-06-03T18:46:53.688486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687439+0000","last_change":"2020-06-03T18:46:53.687141+0000","last_active":"2020-06-03T18:46:53.687439+0000","last_peered":"2020-06-03T18:46:53.687439+0000","last_clean":"2020-06-03T18:46:53.687439+0000","last_became_active":"2020-06-03T18:46:53.686652+0000","last_became_peered":"2020-06-03T18:46:53.686652+0000","last_unstale":"2020-06-03T18:46:53.687439+0000","last_undegraded":"2020-06-03T18:46:53.687439+0000","last_fullsized":"2020-06-03T18:46:53.687439+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.689021+0000","last_change":"2020-06-03T18:46:53.688496+0000","last_active":"2020-06-03T18:46:53.689021+0000","last_peered":"2020-06-03T18:46:53.689021+0000","last_clean":"2020-06-03T18:46:53.689021+0000","last_became_active":"2020-06-03T18:46:53.688119+0000","last_became_peered":"2020-06-03T18:46:53.688119+0000","last_unstale":"2020-06-03T18:46:53.689021+0000","last_undegraded":"2020-06-03T18:46:53.689021+0000","last_fullsized":"2020-06-03T18:46:53.689021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687114+0000","last_change":"2020-06-03T18:46:53.686567+0000","last_active":"2020-06-03T18:46:53.687114+0000","last_peered":"2020-06-03T18:46:53.687114+0000","last_clean":"2020-06-03T18:46:53.687114+0000","last_became_active":"2020-06-03T18:46:53.686026+0000","last_became_peered":"2020-06-03T18:46:53.686026+0000","last_unstale":"2020-06-03T18:46:53.687114+0000","last_undegraded":"2020-06-03T18:46:53.687114+0000","last_fullsized":"2020-06-03T18:46:53.687114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.680966+0000","last_change":"2020-06-03T18:46:50.483314+0000","last_active":"2020-06-03T18:46:53.680966+0000","last_peered":"2020-06-03T18:46:53.680966+0000","last_clean":"2020-06-03T18:46:53.680966+0000","last_became_active":"2020-06-03T18:46:50.481797+0000","last_became_peered":"2020-06-03T18:46:50.481797+0000","last_unstale":"2020-06-03T18:46:53.680966+0000","last_undegraded":"2020-06-03T18:46:53.680966+0000","last_fullsized":"2020-06-03T18:46:53.680966+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:48.571319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:48.571319+0000","last_clean_scrub_stamp":"2020-06-03T18:46:48.571319+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.691107+0000","last_change":"2020-06-03T18:46:53.690778+0000","last_active":"2020-06-03T18:46:53.691107+0000","last_peered":"2020-06-03T18:46:53.691107+0000","last_clean":"2020-06-03T18:46:53.691107+0000","last_became_active":"2020-06-03T18:46:53.690314+0000","last_became_peered":"2020-06-03T18:46:53.690314+0000","last_unstale":"2020-06-03T18:46:53.691107+0000","last_undegraded":"2020-06-03T18:46:53.691107+0000","last_fullsized":"2020-06-03T18:46:53.691107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.685630+0000","last_change":"2020-06-03T18:46:53.684973+0000","last_active":"2020-06-03T18:46:53.685630+0000","last_peered":"2020-06-03T18:46:53.685630+0000","last_clean":"2020-06-03T18:46:53.685630+0000","last_became_active":"2020-06-03T18:46:53.684571+0000","last_became_peered":"2020-06-03T18:46:53.684571+0000","last_unstale":"2020-06-03T18:46:53.685630+0000","last_undegraded":"2020-06-03T18:46:53.685630+0000","last_fullsized":"2020-06-03T18:46:53.685630+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.689695+0000","last_change":"2020-06-03T18:46:53.689695+0000","last_active":"2020-06-03T18:46:53.689695+0000","last_peered":"2020-06-03T18:46:53.689695+0000","last_clean":"2020-06-03T18:46:53.689695+0000","last_became_active":"2020-06-03T18:46:53.688926+0000","last_became_peered":"2020-06-03T18:46:53.688926+0000","last_unstale":"2020-06-03T18:46:53.689695+0000","last_undegraded":"2020-06-03T18:46:53.689695+0000","last_fullsized":"2020-06-03T18:46:53.689695+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687278+0000","last_change":"2020-06-03T18:46:53.686838+0000","last_active":"2020-06-03T18:46:53.687278+0000","last_peered":"2020-06-03T18:46:53.687278+0000","last_clean":"2020-06-03T18:46:53.687278+0000","last_became_active":"2020-06-03T18:46:53.686379+0000","last_became_peered":"2020-06-03T18:46:53.686379+0000","last_unstale":"2020-06-03T18:46:53.687278+0000","last_undegraded":"2020-06-03T18:46:53.687278+0000","last_fullsized":"2020-06-03T18:46:53.687278+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]}]}]}} 2020-06-03T18:46:59.659 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-06-03T18:46:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:46:59.925 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2020-06-03T18:46:59.934 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-03T18:46:59.555626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":91182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.882374"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.688486+0000","last_change":"2020-06-03T18:46:53.688212+0000","last_active":"2020-06-03T18:46:53.688486+0000","last_peered":"2020-06-03T18:46:53.688486+0000","last_clean":"2020-06-03T18:46:53.688486+0000","last_became_active":"2020-06-03T18:46:53.687776+0000","last_became_peered":"2020-06-03T18:46:53.687776+0000","last_unstale":"2020-06-03T18:46:53.688486+0000","last_undegraded":"2020-06-03T18:46:53.688486+0000","last_fullsized":"2020-06-03T18:46:53.688486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687439+0000","last_change":"2020-06-03T18:46:53.687141+0000","last_active":"2020-06-03T18:46:53.687439+0000","last_peered":"2020-06-03T18:46:53.687439+0000","last_clean":"2020-06-03T18:46:53.687439+0000","last_became_active":"2020-06-03T18:46:53.686652+0000","last_became_peered":"2020-06-03T18:46:53.686652+0000","last_unstale":"2020-06-03T18:46:53.687439+0000","last_undegraded":"2020-06-03T18:46:53.687439+0000","last_fullsized":"2020-06-03T18:46:53.687439+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.689021+0000","last_change":"2020-06-03T18:46:53.688496+0000","last_active":"2020-06-03T18:46:53.689021+0000","last_peered":"2020-06-03T18:46:53.689021+0000","last_clean":"2020-06-03T18:46:53.689021+0000","last_became_active":"2020-06-03T18:46:53.688119+0000","last_became_peered":"2020-06-03T18:46:53.688119+0000","last_unstale":"2020-06-03T18:46:53.689021+0000","last_undegraded":"2020-06-03T18:46:53.689021+0000","last_fullsized":"2020-06-03T18:46:53.689021+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687114+0000","last_change":"2020-06-03T18:46:53.686567+0000","last_active":"2020-06-03T18:46:53.687114+0000","last_peered":"2020-06-03T18:46:53.687114+0000","last_clean":"2020-06-03T18:46:53.687114+0000","last_became_active":"2020-06-03T18:46:53.686026+0000","last_became_peered":"2020-06-03T18:46:53.686026+0000","last_unstale":"2020-06-03T18:46:53.687114+0000","last_undegraded":"2020-06-03T18:46:53.687114+0000","last_fullsized":"2020-06-03T18:46:53.687114+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.680966+0000","last_change":"2020-06-03T18:46:50.483314+0000","last_active":"2020-06-03T18:46:53.680966+0000","last_peered":"2020-06-03T18:46:53.680966+0000","last_clean":"2020-06-03T18:46:53.680966+0000","last_became_active":"2020-06-03T18:46:50.481797+0000","last_became_peered":"2020-06-03T18:46:50.481797+0000","last_unstale":"2020-06-03T18:46:53.680966+0000","last_undegraded":"2020-06-03T18:46:53.680966+0000","last_fullsized":"2020-06-03T18:46:53.680966+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:48.571319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:48.571319+0000","last_clean_scrub_stamp":"2020-06-03T18:46:48.571319+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.691107+0000","last_change":"2020-06-03T18:46:53.690778+0000","last_active":"2020-06-03T18:46:53.691107+0000","last_peered":"2020-06-03T18:46:53.691107+0000","last_clean":"2020-06-03T18:46:53.691107+0000","last_became_active":"2020-06-03T18:46:53.690314+0000","last_became_peered":"2020-06-03T18:46:53.690314+0000","last_unstale":"2020-06-03T18:46:53.691107+0000","last_undegraded":"2020-06-03T18:46:53.691107+0000","last_fullsized":"2020-06-03T18:46:53.691107+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.685630+0000","last_change":"2020-06-03T18:46:53.684973+0000","last_active":"2020-06-03T18:46:53.685630+0000","last_peered":"2020-06-03T18:46:53.685630+0000","last_clean":"2020-06-03T18:46:53.685630+0000","last_became_active":"2020-06-03T18:46:53.684571+0000","last_became_peered":"2020-06-03T18:46:53.684571+0000","last_unstale":"2020-06-03T18:46:53.685630+0000","last_undegraded":"2020-06-03T18:46:53.685630+0000","last_fullsized":"2020-06-03T18:46:53.685630+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.689695+0000","last_change":"2020-06-03T18:46:53.689695+0000","last_active":"2020-06-03T18:46:53.689695+0000","last_peered":"2020-06-03T18:46:53.689695+0000","last_clean":"2020-06-03T18:46:53.689695+0000","last_became_active":"2020-06-03T18:46:53.688926+0000","last_became_peered":"2020-06-03T18:46:53.688926+0000","last_unstale":"2020-06-03T18:46:53.689695+0000","last_undegraded":"2020-06-03T18:46:53.689695+0000","last_fullsized":"2020-06-03T18:46:53.689695+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-03T18:46:53.687278+0000","last_change":"2020-06-03T18:46:53.686838+0000","last_active":"2020-06-03T18:46:53.687278+0000","last_peered":"2020-06-03T18:46:53.687278+0000","last_clean":"2020-06-03T18:46:53.687278+0000","last_became_active":"2020-06-03T18:46:53.686379+0000","last_became_peered":"2020-06-03T18:46:53.686379+0000","last_unstale":"2020-06-03T18:46:53.687278+0000","last_undegraded":"2020-06-03T18:46:53.687278+0000","last_fullsized":"2020-06-03T18:46:53.687278+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-03T18:46:52.657626+0000","last_clean_scrub_stamp":"2020-06-03T18:46:52.657626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.381}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.988}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4750000000000001}]}]}]}} 2020-06-03T18:46:59.936 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-06-03T18:46:59.937 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-06-03T18:46:59.937 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-06-03T18:46:59.938 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-06-03T18:47:00.242 INFO:teuthology.orchestra.run.smithi061.stdout: 2020-06-03T18:47:00.262 INFO:teuthology.orchestra.run.smithi061.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-06-03T18:47:00.263 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-06-03T18:47:00.263 INFO:teuthology.run_tasks:Running task rgw... 2020-06-03T18:47:00.285 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-06-03T18:47:00.286 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-06-03T18:47:00.296 INFO:tasks.rgw:Creating data pools 2020-06-03T18:47:00.297 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-06-03T18:47:00.298 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-06-03T18:47:01.478 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' created 2020-06-03T18:47:01.489 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-06-03T18:47:02.485 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-03T18:47:02.507 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-06-03T18:47:03.489 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' created 2020-06-03T18:47:03.503 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-06-03T18:47:04.514 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-03T18:47:04.529 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-06-03T18:47:04.530 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-06-03T18:47:04.793 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' already exists 2020-06-03T18:47:04.803 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-06-03T18:47:05.492 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-03T18:47:05.505 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-06-03T18:47:05.778 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' already exists 2020-06-03T18:47:05.788 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-06-03T18:47:06.493 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-03T18:47:06.504 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-06-03T18:47:06.505 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-06-03T18:47:07.629 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.data' created 2020-06-03T18:47:07.653 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-06-03T18:47:08.645 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-03T18:47:08.664 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-06-03T18:47:09.662 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.index' created 2020-06-03T18:47:09.675 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-06-03T18:47:10.686 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-03T18:47:10.705 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-06-03T18:47:10.706 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-06-03T18:47:10.965 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.data' already exists 2020-06-03T18:47:10.975 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-06-03T18:47:11.665 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-03T18:47:11.678 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-06-03T18:47:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'default.rgw.buckets.index' already exists 2020-06-03T18:47:11.972 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-06-03T18:47:12.669 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-03T18:47:12.682 DEBUG:tasks.rgw:Pools created 2020-06-03T18:47:12.682 INFO:tasks.rgw:Configuring compression type = random 2020-06-03T18:47:12.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-06-03T18:47:12.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-06-03T18:47:12.684 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-06-03T18:47:12.719 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:12.719 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:12.733 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:12.734+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.453 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.453 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.454 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.454 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.455 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.455 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.454+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.455 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.458+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.480 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.481 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.481 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.481 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.482 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.482 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.482 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 10 cannot find current period zonegroup using local zonegroup 2020-06-03T18:47:14.483 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.485 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.485 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.486 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.486 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.482+0000 7f2e20b990c0 10 Creating default zonegroup 2020-06-03T18:47:14.486 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.486+0000 7f2e20b990c0 20 get_system_obj_state: rctx=0x7fffa2df09f0 obj=:.pools.avail state=0x55cb4bd6a4e0 s->prefetch_data=0 2020-06-03T18:47:14.486 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.486+0000 7f2e20b990c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-06-03T18:47:14.501 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.502+0000 7f2e20b990c0 10 WARNING: store->list_zones() returned r=-2 2020-06-03T18:47:14.501 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.502+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.502 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.502+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.507 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.508 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.508 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 10 could not read realm id: (2) No such file or directory 2020-06-03T18:47:14.508 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 10 WARNING: failed to set zone as default, r=-22 2020-06-03T18:47:14.508 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.509 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.510+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.515 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.516 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:14.516 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.516 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 zonegroup default 2020-06-03T18:47:14.517 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 10 Cannot find current period zone using local zone 2020-06-03T18:47:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 10 Using default name default 2020-06-03T18:47:14.518 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 zone default found 2020-06-03T18:47:14.519 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:14.518+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:16.478+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:16.478+0000 7f2e20b990c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:16.478+0000 7f2e20b990c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:16.478+0000 7f2e20b990c0 20 started sync module instance, tier type = 2020-06-03T18:47:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:16.478+0000 7f2e20b990c0 20 started zone id=b75fb9f7-96d9-4833-b88e-92bd7763596b (name=default) with tier type = 2020-06-03T18:47:18.526 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=0 2020-06-03T18:47:18.527 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=1 2020-06-03T18:47:18.527 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=2 2020-06-03T18:47:18.527 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=3 2020-06-03T18:47:18.527 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=4 2020-06-03T18:47:18.527 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=5 2020-06-03T18:47:18.528 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.526+0000 7f2e20b990c0 20 add_watcher() i=6 2020-06-03T18:47:18.528 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.530+0000 7f2e20b990c0 20 add_watcher() i=7 2020-06-03T18:47:18.529 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.530+0000 7f2e20b990c0 2 all 8 watchers are set, enabling cache 2020-06-03T18:47:18.529 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.530+0000 7f2e20b990c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:18.529 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.530+0000 7f2e20b990c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:18.529 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:18.530+0000 7f2dd9feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:22.493 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.494+0000 7f2e20b990c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-06-03T18:47:22.606 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.606+0000 7f2d50ff9700 20 reqs_thread_entry: start 2020-06-03T18:47:22.606 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.606+0000 7f2e20b990c0 20 init_complete bucket index max shards: 11 2020-06-03T18:47:22.606 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.606+0000 7f2d3e7fc700 20 reqs_thread_entry: start 2020-06-03T18:47:22.607 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.606+0000 7f2d3cff9700 20 reqs_thread_entry: start 2020-06-03T18:47:22.631 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.630+0000 7f2dd9feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:22.633 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.634+0000 7f2e20b990c0 20 remove_watcher() i=0 2020-06-03T18:47:22.634 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.634+0000 7f2e20b990c0 2 removed watcher, disabling cache 2020-06-03T18:47:22.642 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.638+0000 7f2e20b990c0 20 remove_watcher() i=1 2020-06-03T18:47:22.644 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.638+0000 7f2e20b990c0 20 remove_watcher() i=2 2020-06-03T18:47:22.645 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.642+0000 7f2e20b990c0 20 remove_watcher() i=3 2020-06-03T18:47:22.645 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.646+0000 7f2e20b990c0 20 remove_watcher() i=4 2020-06-03T18:47:22.647 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.650+0000 7f2e20b990c0 20 remove_watcher() i=5 2020-06-03T18:47:22.650 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.650+0000 7f2e20b990c0 20 remove_watcher() i=6 2020-06-03T18:47:22.652 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.654+0000 7f2e20b990c0 20 remove_watcher() i=7 2020-06-03T18:47:22.662 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T18:47:22.663 DEBUG:tasks.util.rgw: json result: [] 2020-06-03T18:47:22.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:22.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:22.664 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-03T18:47:22.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:22.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:22.760 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.762+0000 7f1d76ecf0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:22.761 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.762+0000 7f1d76ecf0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:22.768 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1d76ecf0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.768 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1ca7fff700 20 reqs_thread_entry: start 2020-06-03T18:47:22.769 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1d76ecf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.770 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1d76ecf0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.770 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1d76ecf0c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:22.770 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.770+0000 7f1d76ecf0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.771 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.774+0000 7f1d76ecf0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.771 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.774+0000 7f1d76ecf0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.771 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.774+0000 7f1d76ecf0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.772 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.774+0000 7f1d76ecf0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.772 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.774+0000 7f1d76ecf0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:22.782 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b75fb9f7-96d9-4833-b88e-92bd7763596b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:22.782 DEBUG:tasks.util.rgw: json result: {'id': 'b75fb9f7-96d9-4833-b88e-92bd7763596b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-06-03T18:47:22.783 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-06-03T18:47:22.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-06-03T18:47:22.784 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-06-03T18:47:22.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:22.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:22.873 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.874+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.882 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.883 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.884 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.878+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.900 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 RGWRados::pool_iterate: got zone_info.b75fb9f7-96d9-4833-b88e-92bd7763596b 2020-06-03T18:47:22.901 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 RGWRados::pool_iterate: got zonegroup_info.6894c940-84bb-46b8-9599-38fa477512b4 2020-06-03T18:47:22.901 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 RGWRados::pool_iterate: got zone_names.default 2020-06-03T18:47:22.901 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-06-03T18:47:22.901 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.903 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.903 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.904 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:22.906 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.906 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.907 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.902+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.907 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:22.907 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.909 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.909 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 10 cannot find current period zonegroup using local zonegroup 2020-06-03T18:47:22.910 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.910 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.910 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.910 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.910 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 zonegroup default 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.906+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 10 Cannot find current period zone using local zone 2020-06-03T18:47:22.911 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.912 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.912 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.912 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:22.912 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.912 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:22.913 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 zone default found 2020-06-03T18:47:22.913 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.913 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.914 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.910+0000 7ff16c9940c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:22.914 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.914+0000 7ff16c9940c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:22.914 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.914+0000 7ff16c9940c0 20 started sync module instance, tier type = 2020-06-03T18:47:22.914 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.914+0000 7ff16c9940c0 20 started zone id=b75fb9f7-96d9-4833-b88e-92bd7763596b (name=default) with tier type = 2020-06-03T18:47:22.930 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=0 2020-06-03T18:47:22.931 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=1 2020-06-03T18:47:22.931 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=2 2020-06-03T18:47:22.931 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=3 2020-06-03T18:47:22.931 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=4 2020-06-03T18:47:22.932 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=5 2020-06-03T18:47:22.932 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.930+0000 7ff16c9940c0 20 add_watcher() i=6 2020-06-03T18:47:22.932 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.934+0000 7ff16c9940c0 20 add_watcher() i=7 2020-06-03T18:47:22.933 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.934+0000 7ff16c9940c0 2 all 8 watchers are set, enabling cache 2020-06-03T18:47:22.933 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.934+0000 7ff16c9940c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:22.933 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.934+0000 7ff16c9940c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:22.933 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:22.934+0000 7ff1257ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:23.017 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.018+0000 7ff08ffff700 20 reqs_thread_entry: start 2020-06-03T18:47:23.017 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.018+0000 7ff08dffb700 20 reqs_thread_entry: start 2020-06-03T18:47:23.017 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.018+0000 7ff16c9940c0 20 init_complete bucket index max shards: 11 2020-06-03T18:47:23.018 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.018+0000 7ff07ffff700 20 reqs_thread_entry: start 2020-06-03T18:47:23.048 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.050+0000 7ff1257ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:23.050 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.050+0000 7ff16c9940c0 20 remove_watcher() i=0 2020-06-03T18:47:23.050 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.050+0000 7ff16c9940c0 2 removed watcher, disabling cache 2020-06-03T18:47:23.053 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.054+0000 7ff16c9940c0 20 remove_watcher() i=1 2020-06-03T18:47:23.056 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.058+0000 7ff16c9940c0 20 remove_watcher() i=2 2020-06-03T18:47:23.059 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.062+0000 7ff16c9940c0 20 remove_watcher() i=3 2020-06-03T18:47:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.066+0000 7ff16c9940c0 20 remove_watcher() i=4 2020-06-03T18:47:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.066+0000 7ff16c9940c0 20 remove_watcher() i=5 2020-06-03T18:47:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.070+0000 7ff16c9940c0 20 remove_watcher() i=6 2020-06-03T18:47:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.074+0000 7ff16c9940c0 20 remove_watcher() i=7 2020-06-03T18:47:23.077 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T18:47:23.078 DEBUG:tasks.util.rgw: json result: [] 2020-06-03T18:47:23.078 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:23.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:23.079 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-03T18:47:23.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:23.160 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:23.169 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.170+0000 7f05cf94a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:23.170 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.170+0000 7f05cf94a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:23.178 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f04fbfff700 20 reqs_thread_entry: start 2020-06-03T18:47:23.179 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:23.180 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:23.180 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:23.181 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:23.181 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:23.181 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:23.182 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:23.182 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:23.182 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.182+0000 7f05cf94a0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:23.183 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:23.186+0000 7f05cf94a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:23.193 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b75fb9f7-96d9-4833-b88e-92bd7763596b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:23.194 DEBUG:tasks.util.rgw: json result: {'id': 'b75fb9f7-96d9-4833-b88e-92bd7763596b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-06-03T18:47:23.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-06-03T18:47:23.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-06-03T18:47:23.195 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-06-03T18:47:23.231 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:23.232 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:23.245 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:23.246+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.702+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.702+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.710+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.710+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.714+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.714 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.714+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.714 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.714+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.734+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.738 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.738 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.738 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 10 cannot find current period zonegroup using local zonegroup 2020-06-03T18:47:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.740 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.740 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.740 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 10 Creating default zonegroup 2020-06-03T18:47:24.741 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 20 get_system_obj_state: rctx=0x7ffcf8089f60 obj=:.pools.avail state=0x5571962b64e0 s->prefetch_data=0 2020-06-03T18:47:24.741 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.738+0000 7f607c70d0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-06-03T18:47:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.758+0000 7f607c70d0c0 10 WARNING: store->list_zones() returned r=-2 2020-06-03T18:47:24.760 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.758+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.760 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.758+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.762+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.766+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.766+0000 7f607c70d0c0 10 could not read realm id: (2) No such file or directory 2020-06-03T18:47:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.766+0000 7f607c70d0c0 10 WARNING: failed to set zone as default, r=-22 2020-06-03T18:47:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.766+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.766+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.770+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.770+0000 7f607c70d0c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.770+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 zonegroup default 2020-06-03T18:47:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 10 Cannot find current period zone using local zone 2020-06-03T18:47:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 10 Using default name default 2020-06-03T18:47:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 zone default found 2020-06-03T18:47:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:24.774+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:26.726 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:26.726+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:26.726 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:26.726+0000 7f607c70d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:26.727 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:26.726+0000 7f607c70d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:26.726+0000 7f607c70d0c0 20 started sync module instance, tier type = 2020-06-03T18:47:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:26.726+0000 7f607c70d0c0 20 started zone id=81df4f53-2842-41f2-87ef-8d10ef93b99b (name=default) with tier type = 2020-06-03T18:47:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=0 2020-06-03T18:47:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=1 2020-06-03T18:47:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=2 2020-06-03T18:47:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=3 2020-06-03T18:47:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=4 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=5 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.777+0000 7f607c70d0c0 20 add_watcher() i=6 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.781+0000 7f607c70d0c0 20 add_watcher() i=7 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.781+0000 7f607c70d0c0 2 all 8 watchers are set, enabling cache 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.781+0000 7f607c70d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.781+0000 7f607c70d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:28.784 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:28.781+0000 7f60357ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:32.746 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.743+0000 7f607c70d0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-06-03T18:47:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.859+0000 7f5f9bfff700 20 reqs_thread_entry: start 2020-06-03T18:47:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.859+0000 7f607c70d0c0 20 init_complete bucket index max shards: 11 2020-06-03T18:47:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.859+0000 7f5f99ffb700 20 reqs_thread_entry: start 2020-06-03T18:47:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.859+0000 7f5f87fff700 20 reqs_thread_entry: start 2020-06-03T18:47:32.892 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.887+0000 7f60357ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:32.895 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.891+0000 7f607c70d0c0 20 remove_watcher() i=0 2020-06-03T18:47:32.895 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.891+0000 7f607c70d0c0 2 removed watcher, disabling cache 2020-06-03T18:47:32.897 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.895+0000 7f607c70d0c0 20 remove_watcher() i=1 2020-06-03T18:47:32.900 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.899+0000 7f607c70d0c0 20 remove_watcher() i=2 2020-06-03T18:47:32.902 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.899+0000 7f607c70d0c0 20 remove_watcher() i=3 2020-06-03T18:47:32.906 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.903+0000 7f607c70d0c0 20 remove_watcher() i=4 2020-06-03T18:47:32.908 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.907+0000 7f607c70d0c0 20 remove_watcher() i=5 2020-06-03T18:47:32.911 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.907+0000 7f607c70d0c0 20 remove_watcher() i=6 2020-06-03T18:47:32.913 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:32.911+0000 7f607c70d0c0 20 remove_watcher() i=7 2020-06-03T18:47:32.927 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T18:47:32.927 DEBUG:tasks.util.rgw: json result: [] 2020-06-03T18:47:32.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:32.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:32.928 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-03T18:47:33.008 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:33.009 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.019+0000 7f444bb270c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:33.022 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.019+0000 7f444bb270c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:33.028 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.023+0000 7f444bb270c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.028 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f43f4ff9700 20 reqs_thread_entry: start 2020-06-03T18:47:33.029 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.029 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.030 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados_obj.operate() r=0 bl.length=805 2020-06-03T18:47:33.030 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.031 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.031 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.031 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.032 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.027+0000 7f444bb270c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.032 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.031+0000 7f444bb270c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:33.041 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"81df4f53-2842-41f2-87ef-8d10ef93b99b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:33.042 DEBUG:tasks.util.rgw: json result: {'id': '81df4f53-2842-41f2-87ef-8d10ef93b99b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-06-03T18:47:33.042 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-06-03T18:47:33.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-06-03T18:47:33.043 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-06-03T18:47:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:33.114 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:33.123 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.119+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.125 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.125 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.126 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.126 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.127 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.127 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.127 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.123+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.145 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 RGWRados::pool_iterate: got zonegroup_info.9aaf91ec-4b12-4c35-b4d4-4e1af67e9f2b 2020-06-03T18:47:33.145 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 RGWRados::pool_iterate: got zone_info.81df4f53-2842-41f2-87ef-8d10ef93b99b 2020-06-03T18:47:33.145 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 RGWRados::pool_iterate: got zone_names.default 2020-06-03T18:47:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-06-03T18:47:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.143+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 10 cannot find current period zonegroup using local zonegroup 2020-06-03T18:47:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.147+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 zonegroup default 2020-06-03T18:47:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 10 Cannot find current period zone using local zone 2020-06-03T18:47:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.155 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.155 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 zone default found 2020-06-03T18:47:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.151+0000 7f0a67d120c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.157 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.155+0000 7f0a67d120c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.157 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.155+0000 7f0a67d120c0 20 started sync module instance, tier type = 2020-06-03T18:47:33.157 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.155+0000 7f0a67d120c0 20 started zone id=81df4f53-2842-41f2-87ef-8d10ef93b99b (name=default) with tier type = 2020-06-03T18:47:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=0 2020-06-03T18:47:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=1 2020-06-03T18:47:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=2 2020-06-03T18:47:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=3 2020-06-03T18:47:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=4 2020-06-03T18:47:33.178 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=5 2020-06-03T18:47:33.178 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=6 2020-06-03T18:47:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 add_watcher() i=7 2020-06-03T18:47:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 2 all 8 watchers are set, enabling cache 2020-06-03T18:47:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:33.180 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a67d120c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:33.180 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.175+0000 7f0a20fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:33.271 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.267+0000 7f098b7fe700 20 reqs_thread_entry: start 2020-06-03T18:47:33.271 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.267+0000 7f09897fa700 20 reqs_thread_entry: start 2020-06-03T18:47:33.271 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.267+0000 7f0a67d120c0 20 init_complete bucket index max shards: 11 2020-06-03T18:47:33.272 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.267+0000 7f097b7fe700 20 reqs_thread_entry: start 2020-06-03T18:47:33.300 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.295+0000 7f0a20fe9700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-03T18:47:33.303 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.299+0000 7f0a67d120c0 20 remove_watcher() i=0 2020-06-03T18:47:33.303 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.299+0000 7f0a67d120c0 2 removed watcher, disabling cache 2020-06-03T18:47:33.306 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.303+0000 7f0a67d120c0 20 remove_watcher() i=1 2020-06-03T18:47:33.309 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.307+0000 7f0a67d120c0 20 remove_watcher() i=2 2020-06-03T18:47:33.312 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.311+0000 7f0a67d120c0 20 remove_watcher() i=3 2020-06-03T18:47:33.315 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.311+0000 7f0a67d120c0 20 remove_watcher() i=4 2020-06-03T18:47:33.318 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.315+0000 7f0a67d120c0 20 remove_watcher() i=5 2020-06-03T18:47:33.320 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.319+0000 7f0a67d120c0 20 remove_watcher() i=6 2020-06-03T18:47:33.322 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.319+0000 7f0a67d120c0 20 remove_watcher() i=7 2020-06-03T18:47:33.331 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T18:47:33.331 DEBUG:tasks.util.rgw: json result: [] 2020-06-03T18:47:33.331 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:33.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-03T18:47:33.332 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-03T18:47:33.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:33.417 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:33.427 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.423+0000 7fbb2ce5d0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-03T18:47:33.428 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.423+0000 7fbb2ce5d0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-03T18:47:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.431+0000 7fba5dffb700 20 reqs_thread_entry: start 2020-06-03T18:47:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.431+0000 7fbb2ce5d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.431+0000 7fbb2ce5d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.431+0000 7fbb2ce5d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados_obj.operate() r=0 bl.length=821 2020-06-03T18:47:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-03T18:47:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-03T18:47:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados->read ofs=0 len=0 2020-06-03T18:47:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:33.435+0000 7fbb2ce5d0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-03T18:47:33.447 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"81df4f53-2842-41f2-87ef-8d10ef93b99b","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:33.447 DEBUG:tasks.util.rgw: json result: {'id': '81df4f53-2842-41f2-87ef-8d10ef93b99b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2020-06-03T18:47:33.448 INFO:tasks.rgw:Starting rgw... 2020-06-03T18:47:33.448 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-06-03T18:47:33.448 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-03T18:47:33.449 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-06-03T18:47:33.450 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-06-03T18:47:33.452 INFO:tasks.rgw.c1.client.0:Started 2020-06-03T18:47:33.453 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-06-03T18:47:33.453 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-03T18:47:33.453 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-03T18:47:33.454 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-03T18:47:33.457 INFO:tasks.rgw.c1.client.1:Started 2020-06-03T18:47:33.457 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-06-03T18:47:33.458 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-03T18:47:33.458 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-06-03T18:47:33.459 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-06-03T18:47:33.489 INFO:tasks.rgw.c2.client.0:Started 2020-06-03T18:47:33.490 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-06-03T18:47:33.490 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-03T18:47:33.491 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-06-03T18:47:33.491 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-06-03T18:47:33.493 INFO:tasks.rgw.c2.client.1:Started 2020-06-03T18:47:33.494 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8000/ 2020-06-03T18:47:33.495 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2020-06-03T18:47:33.526 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:33.526 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:33.529 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2020-06-03T18:47:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T18:47:34.532 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2020-06-03T18:47:34.546 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:34.546 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:34.547 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2020-06-03T18:47:34.548 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2020-06-03T18:47:34.548 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8001/ 2020-06-03T18:47:34.549 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001/ 2020-06-03T18:47:34.606 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:34.606 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:34.611 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:47:34.611 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2020-06-03T18:47:34.611 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi061.front.sepia.ceph.com:8000/ 2020-06-03T18:47:34.612 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000/ 2020-06-03T18:47:34.633 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:34.634 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:34.640 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 15285 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:47:34.640 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2020-06-03T18:47:34.641 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi061.front.sepia.ceph.com:8001/ 2020-06-03T18:47:34.641 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001/ 2020-06-03T18:47:34.695 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:34.696 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:34.700 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:47:34.701 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2020-06-03T18:47:34.702 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-06-03T18:47:35.157 INFO:tasks.rgw_multisite:creating realm.. 2020-06-03T18:47:35.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.159 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:35.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:35.198 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:35.249 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","name":"test-realm","current_period":"ef6973ed-b092-42db-9338-ef2e19a6edba","epoch":1} 2020-06-03T18:47:35.250 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-06-03T18:47:35.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.251 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:35.352 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:47:35.348+0000 7f7aa88290c0 -1 old format 2020-06-03T18:47:35.367 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}} 2020-06-03T18:47:35.368 INFO:tasks.rgw_multisite:creating master zone.. 2020-06-03T18:47:35.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:35.369 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --endpoints http://smithi112.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:35.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:35.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:35.526 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T18:47:35.527 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-06-03T18:47:36.463 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-06-03T18:47:36.483 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-06-03T18:47:37.485 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-06-03T18:47:37.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:37.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:37.500 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:37.549 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:37.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:37.570 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:37.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:37.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:37.573 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:37.610 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:37.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:45.732 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":1,"predecessor_uuid":"ef6973ed-b092-42db-9338-ef2e19a6edba","sync_status":[],"period_map":{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":2} 2020-06-03T18:47:45.746 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-06-03T18:47:45.746 INFO:teuthology.orchestra.run:waiting for 300 2020-06-03T18:47:45.775 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2020-06-03T18:47:45.767+0000 7fbc7affd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 15684) UID: 0 2020-06-03T18:47:45.775 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2020-06-03T18:47:45.767+0000 7fbc9e9160c0 -1 shutting down 2020-06-03T18:47:50.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-06-03T18:47:51.849 INFO:tasks.rgw.c1.client.0:Stopped 2020-06-03T18:47:51.850 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-06-03T18:47:51.851 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-06-03T18:47:51.854 INFO:tasks.rgw.c1.client.0:Started 2020-06-03T18:47:51.854 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2020-06-03T18:47:51.873 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:51.874 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:51.877 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2020-06-03T18:47:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T18:47:52.881 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2020-06-03T18:47:52.899 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:47:52.900 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:47:52.905 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:47:52.905 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2020-06-03T18:47:52.907 INFO:tasks.rgw_multisite:creating admin user.. 2020-06-03T18:47:52.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:52.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:52.913 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:52.996 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:52.997 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:53.176 INFO:teuthology.orchestra.run.smithi112.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-06-03T18:47:53.204 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-06-03T18:47:53.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:53.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:53.206 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:53.250 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:53.251 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:53.379 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","name":"test-realm","current_period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":2} 2020-06-03T18:47:53.380 INFO:tasks.rgw_multisite:creating zone.. 2020-06-03T18:47:53.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:53.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:53.381 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --endpoints http://smithi061.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:53.535 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:53.529+0000 7fc69c8d20c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T18:47:53.535 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:47:53.529+0000 7fc69c8d20c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T18:47:53.548 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T18:47:53.548 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-06-03T18:47:54.555 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-06-03T18:47:54.582 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-06-03T18:47:55.580 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-06-03T18:47:55.600 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:55.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:55.601 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:55.647 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:55.648 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:55.674 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:47:55.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:55.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:47:55.675 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:47:55.755 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:47:55.756 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:47:55.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-06-03T18:48:03.773 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T18:48:03.872 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":2,"predecessor_uuid":"ef6973ed-b092-42db-9338-ef2e19a6edba","sync_status":[],"period_map":{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":2} 2020-06-03T18:48:03.873 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-06-03T18:48:03.874 INFO:teuthology.orchestra.run:waiting for 300 2020-06-03T18:48:03.918 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T18:48:03.913+0000 7f87148d1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 15877) UID: 0 2020-06-03T18:48:03.918 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T18:48:03.913+0000 7f872b7260c0 -1 shutting down 2020-06-03T18:48:06.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-06-03T18:48:09.976 INFO:tasks.rgw.c2.client.0:Stopped 2020-06-03T18:48:09.977 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-06-03T18:48:09.978 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-06-03T18:48:09.982 INFO:tasks.rgw.c2.client.0:Started 2020-06-03T18:48:09.982 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000 2020-06-03T18:48:09.999 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:09.999 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:10.003 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8000: Connection refused 2020-06-03T18:48:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T18:48:11.006 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8000 2020-06-03T18:48:11.020 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:11.020 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:11.025 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:48:11.026 INFO:teuthology.orchestra.run.smithi061.stdout:anonymous 2020-06-03T18:48:11.027 INFO:tasks.rgw_multisite:creating zone.. 2020-06-03T18:48:11.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:11.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:11.028 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --endpoints http://smithi112.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:11.073 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:11.139 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:48:11.131+0000 7f54a1f620c0 0 failed reading obj info from .rgw.root:zone_info.3ee2667c-4516-4846-aa22-24f7de69bd8e: (2) No such file or directory 2020-06-03T18:48:11.139 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T18:48:11.131+0000 7f54a1f620c0 0 WARNING: could not read zone params for zone id=3ee2667c-4516-4846-aa22-24f7de69bd8e name=test-zone2 2020-06-03T18:48:11.150 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T18:48:11.151 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-06-03T18:48:11.486 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-06-03T18:48:11.512 INFO:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-06-03T18:48:12.502 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-06-03T18:48:12.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:12.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:12.520 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:12.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:12.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:12.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-06-03T18:48:12.606 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2020-06-03T18:48:12.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:12.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:12.607 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:12.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:12.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:20.698 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T18:48:20.786 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":3,"predecessor_uuid":"ef6973ed-b092-42db-9338-ef2e19a6edba","sync_status":[],"period_map":{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":2} 2020-06-03T18:48:20.787 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-06-03T18:48:20.787 INFO:teuthology.orchestra.run:waiting for 300 2020-06-03T18:48:20.829 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T18:48:20.823+0000 7f7b897e3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 15687) UID: 0 2020-06-03T18:48:20.829 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T18:48:20.823+0000 7f7ba0e390c0 -1 shutting down 2020-06-03T18:48:24.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-06-03T18:48:26.890 INFO:tasks.rgw.c1.client.1:Stopped 2020-06-03T18:48:26.890 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-03T18:48:26.891 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-03T18:48:26.895 INFO:tasks.rgw.c1.client.1:Started 2020-06-03T18:48:26.895 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2020-06-03T18:48:26.919 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:26.920 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:26.921 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2020-06-03T18:48:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T18:48:27.934 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2020-06-03T18:48:27.951 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:27.951 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:27.956 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T18:48:27.957 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2020-06-03T18:48:27.957 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-06-03T18:48:27.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:27.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:27.959 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --url http://smithi112.front.sepia.ceph.com:8000 --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:28.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:28.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:28.057 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T18:48:28.053+0000 7fcb68ff9700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.55979c05-ad33-4f8a-aebf-51e0a99d9db4.control err=-107 2020-06-03T18:48:28.088 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","name":"test-realm","current_period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":2} 2020-06-03T18:48:28.088 INFO:tasks.rgw_multisite:creating zone.. 2020-06-03T18:48:28.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:28.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--endpoints', 'http://smithi061.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:28.090 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --endpoints http://smithi061.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:28.142 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:28.143 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:48:28.201+0000 7f14a516f0c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T18:48:28.207 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:48:28.201+0000 7f14a516f0c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T18:48:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:48:28.201+0000 7f14a516f0c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T18:48:28.208 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T18:48:28.201+0000 7f14a516f0c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T18:48:28.219 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T18:48:28.220 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-06-03T18:48:28.708 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-06-03T18:48:28.725 INFO:teuthology.orchestra.run.smithi061:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-06-03T18:48:29.721 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-06-03T18:48:29.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:29.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:29.748 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:29.811 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:29.811 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:29.836 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2020-06-03T18:48:29.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:29.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:29.838 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:29.875 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:29.875 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:29.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-06-03T18:48:37.920 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T18:48:38.192 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","epoch":4,"predecessor_uuid":"ef6973ed-b092-42db-9338-ef2e19a6edba","sync_status":[],"period_map":{"id":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":2} 2020-06-03T18:48:38.192 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-06-03T18:48:38.193 INFO:teuthology.orchestra.run:waiting for 300 2020-06-03T18:48:38.236 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T18:48:38.233+0000 7f70ede3c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 15880) UID: 0 2020-06-03T18:48:38.237 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T18:48:38.233+0000 7f71054920c0 -1 shutting down 2020-06-03T18:48:41.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-06-03T18:48:44.295 INFO:tasks.rgw.c2.client.1:Stopped 2020-06-03T18:48:44.295 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-06-03T18:48:44.296 INFO:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-06-03T18:48:44.298 INFO:tasks.rgw.c2.client.1:Started 2020-06-03T18:48:44.299 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2020-06-03T18:48:44.318 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:44.318 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:44.321 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi061.front.sepia.ceph.com port 8001: Connection refused 2020-06-03T18:48:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T18:48:45.324 INFO:teuthology.orchestra.run.smithi061:> curl http://smithi061.front.sepia.ceph.com:8001 2020-06-03T18:48:45.348 INFO:teuthology.orchestra.run.smithi061.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T18:48:45.348 INFO:teuthology.orchestra.run.smithi061.stderr: Dload Upload Total Spent Left Speed 2020-06-03T18:48:45.349 INFO:teuthology.orchestra.run.smithi061.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 26800 2020-06-03T18:48:45.349 INFO:teuthology.orchestra.run.smithi061.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005ed7f08d-115d-test-zone4","HostId":"115d-test-zone4-test-zonegroup"} 2020-06-03T18:48:45.350 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-06-03T18:48:45.350 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-06-03T18:48:45.436 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-06-03T18:48:45.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:45.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T18:48:45.438 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T18:48:45.481 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:45.482 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:45.662 INFO:teuthology.orchestra.run.smithi112.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"9BYJM32QU637KTZJKI55","secret_key":"7RWslnDewVum94QVlsK3raYdDRhUBKmsS2llmO5A"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-06-03T18:48:45.690 INFO:rgw_multi.tests:meta checkpoint 2020-06-03T18:48:45.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:45.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:45.692 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:48:45.772 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:45.989 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:45.990 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:48:45.991 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210125.653449_40.1","last_update":"2020-06-03T18:48:45.653449Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:48:45.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:48:45.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:45.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:45.992 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:48:46.036 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:46.036 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:46.247 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:48:46.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:48:46.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-06-03T18:48:51.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:51.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:51.251 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:48:51.295 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:51.296 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:51.512 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:48:51.513 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:48:56.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:56.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:48:56.517 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:48:56.559 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:48:56.559 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:48:56.786 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:48:56.787 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:01.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:01.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:01.790 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:01.828 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:01.829 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:02.050 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:02.051 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:07.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:07.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:07.053 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:07.095 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:07.095 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:07.316 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:07.344 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:12.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:12.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:12.347 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:12.390 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:12.390 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:12.607 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:12.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:17.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:17.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:17.611 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:17.647 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:17.648 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:17.852 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:17.853 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:22.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:22.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:22.856 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:22.897 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:22.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:23.118 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:23.118 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:28.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:28.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:28.121 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:28.163 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:28.163 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:28.384 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:28.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:33.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:33.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:33.388 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:33.443 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:33.653 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:33.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:38.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:38.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:38.656 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:38.698 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:38.699 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:38.916 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:38.916 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:43.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:43.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:43.919 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:43.963 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:43.963 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:44.184 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:44.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:49.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:49.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:49.191 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:49.233 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:49.233 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:49.458 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:49.459 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:54.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:54.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:54.462 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:54.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:54.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:49:54.744 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:49:54.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:49:59.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:59.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:49:59.754 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:49:59.799 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:49:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:00.028 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:00.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:05.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:05.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:05.032 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:05.074 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:05.075 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:05.300 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:05.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:10.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:10.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:10.305 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:10.348 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:10.349 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:10.568 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:10.569 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:15.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:15.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:15.571 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:15.842 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:15.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:20.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:20.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:20.854 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:20.897 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:20.897 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:21.122 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:21.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:26.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:26.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:26.127 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:26.167 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:26.168 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:26.397 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:26.397 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:31.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:31.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:31.400 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:31.670 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:31.671 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:36.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:36.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:36.673 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:36.712 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:36.713 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:36.925 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:36.925 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210125.653449_40.1 target= 2020-06-03T18:50:41.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:41.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:41.933 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:41.976 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:41.977 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:42.196 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:42.197 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:50:42.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:50:42.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.199 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:42.442 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:42.443 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:50:42.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:50:42.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.445 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:42.763 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:42.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:50:42.764 INFO:rgw_multi.tests:meta checkpoint 2020-06-03T18:50:42.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:42.766 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:42.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:42.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:42.991 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:50:43.025 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210125.653449_40.1","last_update":"2020-06-03T18:48:45.653449Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:50:43.026 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:50:43.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.028 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:43.068 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:43.069 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:43.281 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:43.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:50:43.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:50:43.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.284 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:43.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:43.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:43.526 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:43.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:50:43.527 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:50:43.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:43.528 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:43.844 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210125.653449_40.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:48:45.653449Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:43.845 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:50:43.845 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-06-03T18:50:43.854 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-1 2020-06-03T18:50:47.568 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-2 2020-06-03T18:50:50.754 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-3 2020-06-03T18:50:50.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:50.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:50.828 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:50.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:50.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:51.019 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:50:51.049 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210250.813548_54.1","last_update":"2020-06-03T18:50:50.813548Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:50:51.050 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:50:51.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:51.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:51.052 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:51.092 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:51.092 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:51.310 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210250.813548_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:50.813548Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:51.310 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:50:51.311 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:50:51.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:51.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:51.320 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:51.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:51.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:51.591 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210247.551191_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:47.551191Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:51.592 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1591210250.813548_54.1 target=1_1591210247.551191_41.1 2020-06-03T18:50:56.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:56.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:56.595 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:56.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:56.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:56.846 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210250.813548_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:50.813548Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:56.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:50:56.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:50:56.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:56.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:56.849 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:56.891 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:56.892 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:57.165 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210250.813548_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:50.813548Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:57.165 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:50:57.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-06-03T18:50:57.214 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-4 2020-06-03T18:50:57.240 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-5 2020-06-03T18:50:57.289 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-6 2020-06-03T18:50:57.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:57.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:57.333 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:57.513 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:50:57.546 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210257.317549_67.1","last_update":"2020-06-03T18:50:57.317549Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:50:57.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:50:57.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:57.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:50:57.548 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:50:57.587 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:50:57.588 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:50:57.790 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210257.312149_65.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:57.312149Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:50:57.791 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210257.317549_67.1 target=1_1591210257.312149_65.1 2020-06-03T18:51:02.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:02.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:02.793 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:02.835 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:02.836 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:03.063 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210257.317549_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:57.317549Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:03.064 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:03.064 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:03.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:03.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:03.066 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:03.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:03.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:03.312 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210257.317549_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:57.317549Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:03.313 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:03.313 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:03.314 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:03.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:03.315 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:03.356 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:03.356 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:03.630 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210257.312149_65.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:57.312149Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:03.635 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=0 master=1_1591210257.317549_67.1 target=1_1591210257.312149_65.1 2020-06-03T18:51:08.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:08.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:08.639 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:08.683 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:08.683 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:08.948 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210257.317549_67.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:50:57.317549Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:08.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:09.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:09.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:09.651 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:09.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:09.686 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:09.858 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:51:09.889 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210269.531714_104.1","last_update":"2020-06-03T18:51:09.531714Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:51:09.890 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:51:09.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:09.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:09.892 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:09.933 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:09.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:10.153 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210269.424919_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:09.424919Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:10.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210269.531714_104.1 target=1_1591210269.424919_98.1 2020-06-03T18:51:15.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.157 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:15.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:15.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:15.416 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210269.531714_104.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:09.531714Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:15.417 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:15.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:15.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.419 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:15.460 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:15.461 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:15.667 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210269.531714_104.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:09.531714Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:15.668 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:15.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:15.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:15.670 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:15.711 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:15.986 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210269.531714_104.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:09.531714Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:15.987 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:16.035 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-06-03T18:51:16.036 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-7 2020-06-03T18:51:16.063 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-8 2020-06-03T18:51:16.114 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-9 2020-06-03T18:51:16.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:16.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:16.161 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:16.194 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:16.194 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:16.348 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:51:16.381 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210276.144087_117.1","last_update":"2020-06-03T18:51:16.144087Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:51:16.381 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:51:16.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:16.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:16.383 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:16.424 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:16.425 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:16.645 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210276.138334_115.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:16.138334Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:16.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591210276.144087_117.1 target=1_1591210276.138334_115.1 2020-06-03T18:51:21.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:21.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:21.648 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:21.690 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:21.691 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:21.912 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210276.144087_117.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:16.144087Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:21.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:21.914 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:21.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:21.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:21.915 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:21.958 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:21.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:22.161 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210276.144087_117.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:16.144087Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:22.162 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:22.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:22.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.164 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:22.204 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:22.205 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:22.477 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210276.144087_117.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:16.144087Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:22.478 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:22.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.729 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:22.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:22.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:22.921 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:51:22.951 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:51:22.951 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:51:22.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:22.953 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:22.995 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:22.996 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:23.219 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:23.219 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:23.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:23.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:23.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:23.222 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:23.260 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:23.464 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:23.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:23.465 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:23.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:23.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:23.466 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:23.515 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:23.515 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:23.784 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:23.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:23.809 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-06-03T18:51:23.810 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-10 2020-06-03T18:51:23.841 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-11 2020-06-03T18:51:23.892 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-12 2020-06-03T18:51:24.015 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T18:51:24.011+0000 7fd219ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.017 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T18:51:24.011+0000 7fd223fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.018 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T18:51:24.013+0000 7fcc4b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.018 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T18:51:24.013+0000 7fcc4bfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.031 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T18:51:24.017+0000 7fc3a67fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.031 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T18:51:24.017+0000 7fc3a27f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T18:51:24.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:24.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:24.535 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:24.574 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:24.574 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:24.748 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:51:24.771 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210283.922951_49.1","last_update":"2020-06-03T18:51:23.922951Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:51:24.772 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:51:24.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:24.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:24.774 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:24.824 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:24.825 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:25.007 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210283.922951_49.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:23.922951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:25.007 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:25.008 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:25.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.009 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:25.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:25.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:25.218 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210283.922951_49.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:23.922951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:25.218 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:25.219 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:25.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.220 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:25.263 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:25.263 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:25.541 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210283.922951_49.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:23.922951Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:25.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:25.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.544 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:25.583 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:25.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:25.787 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:25.788 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:25.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:25.789 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:25.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:25.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:26.071 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5","timestamp":"2020-06-03T18:51:24.103871Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:24.142260Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-03T18:51:24.258287Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:51:24.306328Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:26.071 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:26.072 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-10 zones={test-zone2, test-zone1} 2020-06-03T18:51:26.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:26.099 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:26.100 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:26.101 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:26.101 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:26.102 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:26.112 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:26.113 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:26.114 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:26.114 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:26.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:26.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:26.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:26.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:26.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:26.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:26.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:26.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:26.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:27.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:27.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:27.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:27.283 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-10 zones={test-zone2, test-zone1} 2020-06-03T18:51:27.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:27.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:27.285 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:27.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:27.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:27.528 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:27.529 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:27.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:27.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:27.530 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:27.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:27.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:27.825 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5","timestamp":"2020-06-03T18:51:24.138471Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:24.138531Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-03T18:51:24.306290Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:51:24.314718Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:27.825 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:27.826 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-10 zones={test-zone3, test-zone1} 2020-06-03T18:51:27.836 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:27.844 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:27.844 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:27.845 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:27.845 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:27.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:27.852 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:27.853 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:27.853 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:27.854 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:27.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:27.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:28.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:28.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:28.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:28.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:28.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:28.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:28.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:28.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:28.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:28.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:29.028 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-10 zones={test-zone3, test-zone1} 2020-06-03T18:51:29.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.030 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:29.068 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:29.069 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:29.277 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.6.5"},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:29.282 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:29.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.284 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:29.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:29.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:29.617 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.24.5","timestamp":"2020-06-03T18:51:23.948612Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:23.998449Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-03T18:51:24.096300Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:51:24.046873Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:29.617 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-10 2020-06-03T18:51:29.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.619 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:29.699 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:29.699 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:29.914 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.18.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:29.915 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-11 2020-06-03T18:51:29.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:29.917 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:29.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:29.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:30.186 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:24.310385Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-03T18:51:24.314729Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5","timestamp":"2020-06-03T18:51:24.461502Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-03T18:51:24.510313Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:30.187 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-11 2020-06-03T18:51:30.187 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-11 zones={test-zone1, test-zone2} 2020-06-03T18:51:30.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:30.206 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:30.207 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:30.207 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:30.208 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:30.208 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:30.220 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:30.221 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:30.221 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:30.222 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:30.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:30.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:30.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:30.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:30.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:30.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:30.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:30.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:30.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:31.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:31.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:31.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:31.370 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-11 zones={test-zone1, test-zone2} 2020-06-03T18:51:31.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:31.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:31.373 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:31.412 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:31.639 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.18.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:31.639 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-11 2020-06-03T18:51:31.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:31.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:31.642 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:31.716 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:31.716 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:31.929 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:24.314387Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-03T18:51:24.310375Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.5","timestamp":"2020-06-03T18:51:24.510308Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-03T18:51:24.506739Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:31.929 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-11 2020-06-03T18:51:31.930 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-11 zones={test-zone3, test-zone2} 2020-06-03T18:51:31.949 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:31.963 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:31.963 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:31.964 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:31.964 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:31.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:31.978 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:31.979 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:31.979 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:31.980 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:32.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:32.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:32.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:32.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:32.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:32.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:32.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:32.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:32.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:32.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:32.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:33.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:33.142 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-11 zones={test-zone3, test-zone2} 2020-06-03T18:51:33.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:33.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:33.144 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:33.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:33.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:33.380 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.17.5"},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:33.381 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-12 2020-06-03T18:51:33.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:33.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:33.383 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:33.470 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:33.471 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:33.681 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-03T18:51:24.478342Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-03T18:51:24.478312Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-03T18:51:24.674290Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:24.672858Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:33.682 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-12 2020-06-03T18:51:33.682 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-12 zones={test-zone1, test-zone3} 2020-06-03T18:51:33.694 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:33.701 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:33.702 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:33.702 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:33.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:33.709 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:33.709 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:33.710 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:33.710 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:33.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:33.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:33.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:34.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:34.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:34.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:34.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:34.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:34.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:34.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:34.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:34.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:34.860 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-12 zones={test-zone1, test-zone3} 2020-06-03T18:51:34.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:34.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:34.862 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:34.900 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:34.900 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:35.109 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.17.5"},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:35.109 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-12 2020-06-03T18:51:35.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:35.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:35.111 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:35.408 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-03T18:51:24.478640Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-03T18:51:24.433108Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-03T18:51:24.670274Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:24.722349Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:35.409 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-12 2020-06-03T18:51:35.410 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-12 zones={test-zone2, test-zone3} 2020-06-03T18:51:35.422 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:35.430 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:35.431 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:35.432 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:35.432 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:35.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:35.443 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-03T18:51:35.444 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-03T18:51:35.445 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-03T18:51:35.445 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:35.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:35.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:35.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-03T18:51:35.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:35.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:35.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-03T18:51:36.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:36.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:36.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-03T18:51:36.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:36.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:36.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:36.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-12 zones={test-zone2, test-zone3} 2020-06-03T18:51:36.588 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-06-03T18:51:36.589 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-13 2020-06-03T18:51:36.619 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-14 2020-06-03T18:51:36.669 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-15 2020-06-03T18:51:36.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:36.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:36.864 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:36.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:36.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:37.056 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:51:37.079 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210296.695551_73.1","last_update":"2020-06-03T18:51:36.695551Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:51:37.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:51:37.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.082 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:37.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:37.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:37.325 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210296.695551_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:36.695551Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:37.326 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:51:37.326 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:51:37.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.328 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:37.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:37.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:37.541 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210296.695551_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:36.695551Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:37.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:51:37.542 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:51:37.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.544 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:37.852 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210296.695551_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:51:36.695551Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:51:37.853 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:51:37.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:37.854 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:37.899 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:37.899 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:38.112 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.12.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:38.113 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:38.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:38.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:38.114 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:38.160 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:38.161 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:38.403 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.878270Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:38.404 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:38.404 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-13 zones={test-zone2, test-zone1} 2020-06-03T18:51:38.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:38.462 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:38.463 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:38.475 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:38.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:38.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:38.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:38.816 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-13 zones={test-zone2, test-zone1} 2020-06-03T18:51:38.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:38.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:38.818 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:39.063 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.12.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:39.064 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:39.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:39.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:39.065 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:39.149 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:39.149 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:39.360 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.878387Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:39.361 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:39.362 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-13 zones={test-zone3, test-zone1} 2020-06-03T18:51:39.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:39.422 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:39.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:39.431 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:39.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:39.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:39.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:39.764 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-13 zones={test-zone3, test-zone1} 2020-06-03T18:51:39.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:39.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:39.766 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:39.801 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:39.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:40.007 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.12.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:40.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:40.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.009 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:40.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:40.057 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:40.332 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.720201Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:40.333 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:40.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.335 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:40.602 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:40.603 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:51:40.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:40.605 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:40.653 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:40.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:40.870 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906374Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:40.871 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:51:40.871 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-14 zones={test-zone1, test-zone2} 2020-06-03T18:51:40.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:40.930 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:40.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:40.940 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:40.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:41.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:41.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:41.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-14 zones={test-zone1, test-zone2} 2020-06-03T18:51:41.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:41.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:41.279 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:41.317 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:41.318 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:41.536 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:41.537 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:51:41.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:41.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:41.540 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:41.577 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:41.578 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:41.801 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906372Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:41.802 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:51:41.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-14 zones={test-zone3, test-zone2} 2020-06-03T18:51:41.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:41.866 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:41.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:41.875 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:41.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:41.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:42.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:42.226 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-14 zones={test-zone3, test-zone2} 2020-06-03T18:51:42.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:42.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:42.229 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:42.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:42.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:42.466 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:42.466 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:51:42.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:42.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:42.468 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:42.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:42.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:42.773 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5","timestamp":"2020-06-03T18:51:37.037137Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:42.774 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:51:42.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-15 zones={test-zone1, test-zone3} 2020-06-03T18:51:42.786 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:42.834 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:42.835 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:42.843 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:42.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:42.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:43.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:43.186 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-15 zones={test-zone1, test-zone3} 2020-06-03T18:51:43.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:43.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:43.188 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:43.227 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:43.227 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:43.421 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:43.421 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:51:43.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:43.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:43.424 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:43.711 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5","timestamp":"2020-06-03T18:51:37.038743Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:43.712 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:51:43.712 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-15 zones={test-zone2, test-zone3} 2020-06-03T18:51:43.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:51:43.774 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:43.775 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:51:43.784 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-03T18:51:43.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:43.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:44.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-03T18:51:44.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-15 zones={test-zone2, test-zone3} 2020-06-03T18:51:44.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:44.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:44.144 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:44.179 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:44.180 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:44.375 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:51:44.376 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:51:44.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:44.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:44.377 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:44.422 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:44.660 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.878270Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:44.661 WARNING:rgw_multi.tests:bucket rhmmtn-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.14.5 target=00000000001.12.5 2020-06-03T18:51:49.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:49.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:49.663 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:49.707 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:49.707 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:49.961 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.878270Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:49.962 WARNING:rgw_multi.tests:bucket rhmmtn-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.14.5 target=00000000001.12.5 2020-06-03T18:51:54.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:54.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:51:54.965 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:51:55.011 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:51:55.012 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:51:55.249 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-03T18:51:36.878270Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:51:55.250 WARNING:rgw_multi.tests:bucket rhmmtn-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.14.5 target=00000000001.12.5 2020-06-03T18:52:00.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.252 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:00.299 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:00.300 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:00.539 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-06-03T18:51:36.720201Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:00.539 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:52:00.540 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-13 zones={test-zone2, test-zone1} 2020-06-03T18:52:00.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:00.599 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:00.672 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-13 zones={test-zone2, test-zone1} 2020-06-03T18:52:00.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.675 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:00.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:00.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:00.922 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:00.923 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:52:00.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:00.924 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:01.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:01.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:01.226 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-06-03T18:51:36.720201Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:01.226 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:52:01.227 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-13 zones={test-zone3, test-zone1} 2020-06-03T18:52:01.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:01.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:01.361 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-13 zones={test-zone3, test-zone1} 2020-06-03T18:52:01.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.363 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:01.397 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:01.398 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:01.596 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:01.597 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:52:01.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.598 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:01.647 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:01.647 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:01.909 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.5","timestamp":"2020-06-03T18:51:36.720201Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:01.909 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-13 2020-06-03T18:52:01.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:01.930 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:01.968 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:01.969 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:02.191 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:02.192 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:52:02.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:02.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:02.193 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:02.238 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:02.238 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:02.465 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906374Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:02.486 WARNING:rgw_multi.tests:bucket rhmmtn-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.13.5 target=00000000001.9.5 2020-06-03T18:52:07.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:07.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:07.489 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:07.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:07.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:07.752 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906374Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:07.752 WARNING:rgw_multi.tests:bucket rhmmtn-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.13.5 target=00000000001.9.5 2020-06-03T18:52:12.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:12.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:12.755 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:12.798 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:12.799 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:13.028 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906374Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:13.029 WARNING:rgw_multi.tests:bucket rhmmtn-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.13.5 target=00000000001.9.5 2020-06-03T18:52:18.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:18.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:18.031 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:18.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:18.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:18.302 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5","timestamp":"2020-06-03T18:51:36.906374Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:18.303 WARNING:rgw_multi.tests:bucket rhmmtn-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.13.5 target=00000000001.9.5 2020-06-03T18:52:23.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.306 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:23.366 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:23.580 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5","timestamp":"2020-06-03T18:51:36.765366Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:23.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:52:23.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-14 zones={test-zone1, test-zone2} 2020-06-03T18:52:23.597 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:23.648 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:23.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-14 zones={test-zone1, test-zone2} 2020-06-03T18:52:23.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.724 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:23.762 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:23.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:23.983 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:23.983 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:52:23.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:23.985 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:24.033 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:24.034 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:24.234 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5","timestamp":"2020-06-03T18:51:36.765366Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:24.235 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-14 2020-06-03T18:52:24.236 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-14 zones={test-zone3, test-zone2} 2020-06-03T18:52:24.248 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:24.298 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:24.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-14 zones={test-zone3, test-zone2} 2020-06-03T18:52:24.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:24.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:24.389 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:24.427 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:24.427 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:24.618 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:24.619 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:52:24.619 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:24.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:24.621 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:24.704 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:24.906 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.8.5","timestamp":"2020-06-03T18:51:36.819983Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:24.907 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:52:24.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-15 zones={test-zone1, test-zone3} 2020-06-03T18:52:24.918 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:24.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:25.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-15 zones={test-zone1, test-zone3} 2020-06-03T18:52:25.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.031 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:25.069 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:25.070 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:25.280 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:25.281 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:52:25.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.283 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:25.327 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:25.569 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.8.5","timestamp":"2020-06-03T18:51:36.819983Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:25.569 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-15 2020-06-03T18:52:25.570 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-15 zones={test-zone2, test-zone3} 2020-06-03T18:52:25.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:52:25.630 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:52:25.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-15 zones={test-zone2, test-zone3} 2020-06-03T18:52:25.709 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-06-03T18:52:25.710 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-16 2020-06-03T18:52:25.736 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-17 2020-06-03T18:52:25.792 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-18 2020-06-03T18:52:25.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:25.904 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:25.935 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:25.935 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:26.100 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:52:26.121 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210345.893519_106.1","last_update":"2020-06-03T18:52:25.893519Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:52:26.122 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:52:26.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.124 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:26.161 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:26.162 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:26.391 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210345.893519_106.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:52:25.893519Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:52:26.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:52:26.392 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:52:26.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.394 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:26.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:26.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:26.638 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210345.893519_106.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:52:25.893519Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:52:26.638 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:52:26.639 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:52:26.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:26.640 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:26.683 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:26.683 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:26.958 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210345.893519_106.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:52:25.893519Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:52:26.959 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:52:27.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.013 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:27.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:27.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:27.246 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.13.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:27.246 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:52:27.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.248 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:27.294 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:27.295 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:27.506 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10","timestamp":"2020-06-03T18:52:27.098390Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:27.506 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:52:27.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.508 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:27.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:27.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:27.748 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:27.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:52:27.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:27.750 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:27.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:27.833 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:28.051 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10","timestamp":"2020-06-03T18:52:27.102393Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:28.052 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.19.9 target=00000000002.13.10 2020-06-03T18:52:33.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:33.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:33.055 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:33.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:33.102 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:33.328 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10","timestamp":"2020-06-03T18:52:27.102393Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:33.328 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.19.9 target=00000000002.13.10 2020-06-03T18:52:38.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:38.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:38.333 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:38.590 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10","timestamp":"2020-06-03T18:52:27.102393Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:38.591 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.19.9 target=00000000002.13.10 2020-06-03T18:52:43.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:43.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:43.594 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:43.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:43.636 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:43.868 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10","timestamp":"2020-06-03T18:52:27.102393Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:43.869 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.19.9 target=00000000002.13.10 2020-06-03T18:52:48.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:48.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:48.876 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:48.922 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:48.922 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:49.148 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000006.21.9","timestamp":"2020-06-03T18:52:26.973818Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:49.149 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:52:49.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:49.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:49.150 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:49.192 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:49.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:49.391 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:52:49.392 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:52:49.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:49.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:49.394 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:49.440 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:49.440 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:49.715 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.19.9","timestamp":"2020-06-03T18:52:26.973818Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:49.715 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.21.9 target=00000000005.19.9 2020-06-03T18:52:54.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:54.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:52:54.718 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:52:54.764 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:52:54.765 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:52:55.036 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.19.9","timestamp":"2020-06-03T18:52:26.973818Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:52:55.037 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.21.9 target=00000000005.19.9 2020-06-03T18:53:00.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:00.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:00.041 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:00.084 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:00.084 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:00.349 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.19.9","timestamp":"2020-06-03T18:52:26.973818Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:00.349 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.21.9 target=00000000005.19.9 2020-06-03T18:53:05.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.353 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:05.670 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.21.9","timestamp":"2020-06-03T18:52:26.973818Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:05.670 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:53:05.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.723 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:05.766 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:05.767 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:05.970 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.9.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:53:05.970 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:53:05.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:05.972 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:06.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:06.019 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:06.209 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10","timestamp":"2020-06-03T18:53:05.758442Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:06.210 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:53:06.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:06.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:06.211 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:06.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:06.254 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:06.479 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.13.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:53:06.480 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:53:06.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:06.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:06.481 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:06.523 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:06.524 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:06.731 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10","timestamp":"2020-06-03T18:53:05.758442Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:06.731 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.13.9 target=00000000002.9.10 2020-06-03T18:53:11.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:11.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:11.735 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:11.780 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:11.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:12.018 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10","timestamp":"2020-06-03T18:53:05.758442Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:12.019 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.13.9 target=00000000002.9.10 2020-06-03T18:53:17.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:17.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:17.027 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:17.073 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:17.074 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:17.301 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10","timestamp":"2020-06-03T18:53:05.758442Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:17.302 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.13.9 target=00000000002.9.10 2020-06-03T18:53:22.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:22.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:22.305 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:22.350 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:22.351 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:22.566 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10","timestamp":"2020-06-03T18:53:05.758442Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:22.566 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.13.9 target=00000000002.9.10 2020-06-03T18:53:27.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:27.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:27.569 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:27.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:27.617 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:27.837 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000005.15.9","timestamp":"2020-06-03T18:53:05.680491Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:27.838 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:53:27.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:27.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:27.892 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:27.931 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:27.931 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:28.115 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.25.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:53:28.115 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:53:28.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.117 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:28.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:28.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:28.398 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000002.25.10","timestamp":"2020-06-03T18:53:27.946359Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:28.398 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:53:28.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.400 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:28.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:28.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:28.647 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:53:28.647 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:53:28.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:28.649 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:28.694 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:28.694 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:28.935 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000002.25.10","timestamp":"2020-06-03T18:53:27.946382Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:28.936 WARNING:rgw_multi.tests:bucket rhmmtn-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.31.9 target=00000000002.25.10 2020-06-03T18:53:33.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:33.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:33.939 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:34.231 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000002.25.10","timestamp":"2020-06-03T18:53:27.946382Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:34.232 WARNING:rgw_multi.tests:bucket rhmmtn-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.31.9 target=00000000002.25.10 2020-06-03T18:53:39.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:39.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:39.235 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:39.507 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000002.25.10","timestamp":"2020-06-03T18:53:27.946382Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:39.508 WARNING:rgw_multi.tests:bucket rhmmtn-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.31.9 target=00000000002.25.10 2020-06-03T18:53:44.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:44.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:44.511 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:44.556 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:44.557 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:44.787 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000002.25.10","timestamp":"2020-06-03T18:53:27.946382Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:44.788 WARNING:rgw_multi.tests:bucket rhmmtn-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000005.31.9 target=00000000002.25.10 2020-06-03T18:53:49.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:49.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:49.791 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:49.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:49.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:50.071 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.9","timestamp":"2020-06-03T18:53:27.849331Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:50.072 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:53:51.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:51.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:51.843 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:51.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:51.880 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:52.058 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.45.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:53:52.059 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:53:52.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:52.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:52.062 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:52.108 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:52.109 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:52.344 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.342317Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.10","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:52.345 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000009.45.9 target=00000000008.33.10 2020-06-03T18:53:57.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:57.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:53:57.348 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:53:57.394 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:53:57.394 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:53:57.634 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.342317Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.10","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:53:57.635 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000009.45.9 target=00000000008.33.10 2020-06-03T18:54:02.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:02.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:02.638 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:02.683 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:02.684 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:02.923 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.342317Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.10","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:02.924 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000009.45.9 target=00000000008.33.10 2020-06-03T18:54:07.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:07.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:07.927 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:07.974 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:07.974 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:08.222 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.342317Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.10","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:08.223 WARNING:rgw_multi.tests:bucket rhmmtn-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000009.45.9 target=00000000008.33.10 2020-06-03T18:54:13.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.226 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:13.271 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:13.272 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:13.519 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.342317Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000009.45.9","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:13.519 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:54:13.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.521 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:13.565 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:13.565 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:13.764 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.45.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:13.764 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:54:13.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:13.767 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:13.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:14.085 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.338350Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000009.45.9","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:14.085 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:54:14.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.087 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:14.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:14.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:14.381 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.45.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:14.382 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:54:14.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.385 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:14.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:14.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:14.704 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.35.5","timestamp":"2020-06-03T18:53:50.207405Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.45.9","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:14.705 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:54:14.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:14.707 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:14.787 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:14.787 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:15.015 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.35.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:15.016 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:54:15.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.017 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:15.062 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:15.062 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.084158Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000010.33.5","timestamp":"2020-06-03T18:53:50.414110Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000007.35.9","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:15.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:54:15.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.298 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:15.340 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:15.340 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:15.564 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.19.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.35.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:15.565 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:54:15.566 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.567 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:15.613 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:15.613 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:15.836 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.084158Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000010.33.5","timestamp":"2020-06-03T18:53:50.414110Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000007.35.9","timestamp":"2020-06-03T18:53:50.498763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:15.836 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:54:15.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:15.843 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:15.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:15.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:16.099 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.21.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.53.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:16.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:54:16.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.102 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:16.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:16.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:16.407 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-06-03T18:53:50.084158Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000014.53.5","timestamp":"2020-06-03T18:53:50.613128Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:16.407 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:54:16.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.409 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:16.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:16.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:16.686 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.21.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.53.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:16.687 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:54:16.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:16.688 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:16.965 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-06-03T18:53:50.084158Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9","timestamp":"2020-06-03T18:53:50.293490Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000014.53.5","timestamp":"2020-06-03T18:53:50.613128Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:16.966 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:54:16.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone2, test-zone1} 2020-06-03T18:54:16.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:54:16.988 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:16.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:16.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:16.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:16.990 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:54:16.997 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:16.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:16.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:16.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:17.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:17.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:17.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:17.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:17.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:17.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:17.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:17.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:17.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:17.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:17.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:17.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:18.007 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone2, test-zone1} 2020-06-03T18:54:18.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone3, test-zone1} 2020-06-03T18:54:18.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:54:18.029 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:18.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:18.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:18.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:18.031 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:54:18.036 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:18.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:18.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:18.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:18.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:18.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:18.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:18.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:18.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:18.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:18.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:18.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:18.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:18.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:18.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:19.032 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone3, test-zone1} 2020-06-03T18:54:19.033 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone3, test-zone2} 2020-06-03T18:54:19.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:54:19.050 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:19.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:19.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:19.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:19.052 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:54:19.060 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:54:19.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:54:19.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:54:19.062 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:54:19.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:19.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:19.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:54:19.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:19.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:19.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:54:19.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:19.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:19.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:54:19.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:19.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:19.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:54:20.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone3, test-zone2} 2020-06-03T18:54:20.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:20.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:20.071 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:20.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:20.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:20.329 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.55.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000006.19.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:20.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:54:20.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:20.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:20.331 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:20.377 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:20.377 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:20.627 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938350Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9","timestamp":"2020-06-03T18:53:05.680491Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:20.627 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.19.10 target=00000000004.15.9 2020-06-03T18:54:25.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:25.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:25.634 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:25.682 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:25.682 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:25.926 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938350Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000004.15.9","timestamp":"2020-06-03T18:53:05.680491Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:25.927 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.19.10 target=00000000004.15.9 2020-06-03T18:54:30.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:30.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:30.931 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:30.979 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:30.980 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:31.226 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938350Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:31.226 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:54:31.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:31.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:31.228 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:31.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:31.482 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.55.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:54:31.483 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:54:31.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:31.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:31.485 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:31.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:31.568 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:31.786 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:31.787 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:54:36.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:36.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:36.790 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:36.833 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:36.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:37.062 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:37.063 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:54:42.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:42.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:42.065 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:42.112 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:42.112 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:42.345 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:42.345 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:54:47.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:47.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:47.349 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:47.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:47.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:47.611 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:47.612 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:54:52.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:52.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:52.615 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:52.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:52.888 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:52.889 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:54:57.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:57.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:54:57.891 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:54:57.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:54:57.938 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:54:58.157 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000006.19.10","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:54:58.158 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone3 behind zone test-zone1: shard=5 master=00000000007.21.9 target=00000000006.19.10 2020-06-03T18:55:03.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.161 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:03.204 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:03.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:03.430 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.938298Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:03.431 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:03.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.433 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:03.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:03.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:03.723 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.55.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.19.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:03.723 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:03.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:03.725 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:03.771 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:03.771 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:04.049 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.55.5","timestamp":"2020-06-03T18:53:50.805745Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.19.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:04.050 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:04.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.052 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:04.133 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:04.133 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:04.354 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.41.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:04.355 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:04.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.356 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:04.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:04.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:04.623 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000003.41.9","timestamp":"2020-06-03T18:53:50.683614Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-03T18:53:51.004703Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000008.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:04.623 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:04.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.625 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:04.669 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:04.670 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:04.895 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.41.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:04.896 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:04.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:04.898 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:04.940 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:04.941 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:05.171 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000003.41.9","timestamp":"2020-06-03T18:53:50.683614Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-03T18:53:51.004703Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000008.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:05.172 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:05.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.173 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:05.251 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:05.468 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.59.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:05.468 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:05.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.470 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:05.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:05.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:05.767 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000003.59.9","timestamp":"2020-06-03T18:53:50.683614Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000014.37.5","timestamp":"2020-06-03T18:53:51.196988Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:05.767 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:05.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:05.769 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:05.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:06.057 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.59.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:06.058 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:06.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:06.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:06.059 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:06.108 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:06.109 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:06.353 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000003.59.9","timestamp":"2020-06-03T18:53:50.683614Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:50.888285Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000014.37.5","timestamp":"2020-06-03T18:53:51.196988Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:06.354 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:06.355 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone2, test-zone1} 2020-06-03T18:55:06.367 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:06.375 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:06.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:06.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:06.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:06.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:06.384 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:06.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:06.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:06.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:06.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:06.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:06.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:06.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:06.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:06.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:06.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:06.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:07.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:07.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:07.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:07.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:07.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone2, test-zone1} 2020-06-03T18:55:07.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone3, test-zone1} 2020-06-03T18:55:07.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:07.421 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:07.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:07.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:07.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:07.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:07.429 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:07.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:07.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:07.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:07.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:07.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:07.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:07.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:07.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:07.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:07.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:07.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:08.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:08.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:08.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:08.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:08.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone3, test-zone1} 2020-06-03T18:55:08.447 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone3, test-zone2} 2020-06-03T18:55:08.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:08.464 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:08.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:08.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:08.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:08.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:08.476 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:08.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:08.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:08.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:08.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:08.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:08.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:08.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:08.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:08.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:08.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:08.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:09.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:09.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:09.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:09.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:09.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone3, test-zone2} 2020-06-03T18:55:09.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.468 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:09.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:09.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:09.717 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.65.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:09.717 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:09.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.719 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:09.765 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:09.766 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:09.997 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"odOy-VX0HKRMHFxAA0yseey.h30ZTV2","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-06-03T18:53:51.378855Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:51.942355Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000007.65.9","timestamp":"2020-06-03T18:53:51.660062Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:09.998 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:09.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:09.999 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:10.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:10.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:10.255 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.65.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:10.256 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:10.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.258 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:10.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:10.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:10.575 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"odOy-VX0HKRMHFxAA0yseey.h30ZTV2","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-06-03T18:53:51.378855Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:51.938331Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000007.65.9","timestamp":"2020-06-03T18:53:51.660062Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:10.576 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:10.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.578 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:10.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:10.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:10.873 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.65.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:10.874 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:10.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:10.877 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:11.193 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.29.5","timestamp":"2020-06-03T18:53:51.378855Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:51.451676Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.65.9","timestamp":"2020-06-03T18:53:51.660062Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:11.194 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:11.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.197 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:11.280 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:11.280 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:11.506 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.21.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:11.506 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:11.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.509 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:11.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:11.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:11.769 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-06-03T18:53:51.267029Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-03T18:53:51.578293Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000007.47.9","timestamp":"2020-06-03T18:53:51.660062Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:11.770 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:11.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:11.772 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:11.811 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:11.811 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:12.039 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.21.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.34.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.47.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:12.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:12.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.041 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:12.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:12.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:12.317 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000003.21.9","timestamp":"2020-06-03T18:53:51.267029Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":2},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-03T18:53:51.578293Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000007.47.9","timestamp":"2020-06-03T18:53:51.660062Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:12.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:12.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.320 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:12.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:12.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:12.599 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.20.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.24.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:12.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:12.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.602 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:12.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:12.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:12.908 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:51.267029Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9","timestamp":"2020-06-03T18:53:51.451676Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000016.79.5","timestamp":"2020-06-03T18:53:51.777877Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:12.909 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:12.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:12.912 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:13.203 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.20.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.24.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:13.204 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:13.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:13.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:13.206 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:13.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:13.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:13.499 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:53:51.267029Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9","timestamp":"2020-06-03T18:53:51.451676Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000016.79.5","timestamp":"2020-06-03T18:53:51.777877Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:13.499 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:13.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone2, test-zone1} 2020-06-03T18:55:13.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:13.517 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:13.517 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:13.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:13.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:13.518 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:13.527 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:13.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:13.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:13.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:13.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:13.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:13.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:13.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:13.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:13.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:13.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:14.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:14.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:14.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:14.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:14.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:14.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone2, test-zone1} 2020-06-03T18:55:14.538 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone3, test-zone1} 2020-06-03T18:55:14.551 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:14.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:14.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:14.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:14.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:14.560 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:14.567 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:14.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:14.568 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:14.569 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:14.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:14.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:14.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:14.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:14.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:14.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:14.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:15.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:15.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:15.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:15.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:15.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:15.543 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone3, test-zone1} 2020-06-03T18:55:15.544 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone3, test-zone2} 2020-06-03T18:55:15.556 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:15.563 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:15.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:15.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:15.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:15.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:15.571 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:15.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:15.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:15.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:15.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:15.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:15.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:15.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:15.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:15.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:16.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:16.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:16.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:16.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:16.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:16.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:16.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone3, test-zone2} 2020-06-03T18:55:17.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.223 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:17.429 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.42.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:17.429 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:17.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.431 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:17.476 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:17.477 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:17.700 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000011.42.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000010.70.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:17.701 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:17.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.703 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:17.744 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:17.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:17.960 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.42.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:17.960 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:17.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:17.963 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:18.049 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:18.049 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:18.291 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000011.42.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000010.70.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:18.292 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:18.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.294 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:18.335 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:18.335 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:18.540 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.42.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:18.540 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:18.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.542 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:18.587 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:18.587 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:18.856 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.42.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.70.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:18.857 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-16 2020-06-03T18:55:18.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:18.859 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:18.943 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:18.944 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:19.176 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.52.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:19.177 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:55:19.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.179 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:19.226 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:19.226 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:19.442 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000011.40.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.52.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:19.442 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:55:19.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.444 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:19.486 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:19.487 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:19.707 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.22.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.52.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:19.708 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:55:19.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.711 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:19.756 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:19.757 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:19.982 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000011.40.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000008.52.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:19.983 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-16 2020-06-03T18:55:19.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:19.985 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:20.065 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:20.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:20.269 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.82.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:20.269 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:55:20.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.271 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:20.356 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:20.357 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:20.587 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000015.82.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:20.588 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:55:20.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.590 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:20.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:20.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:20.886 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.82.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:20.887 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:55:20.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:20.888 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:20.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:21.185 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"u.lqQLv5tV9r-5SNyj9jbfaVEmnmUy6","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-06-03T18:55:16.649888Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"sXVjLdZ34BhU6eJQAx.RyzqRGax9ObP","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5","timestamp":"2020-06-03T18:55:16.721392Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"hbLip9672sWO2.N8SO5u7hv6n2mnhHG","ns":""},"count":1},"inc_marker":{"position":"00000000015.82.5","timestamp":"2020-06-03T18:55:16.797428Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:21.186 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-16 2020-06-03T18:55:21.186 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone2, test-zone1} 2020-06-03T18:55:21.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:21.205 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:21.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:21.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:21.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:21.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:21.214 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:21.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:21.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:21.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:21.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:21.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:21.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:21.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:21.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:21.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:21.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:22.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:22.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:22.232 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone2, test-zone1} 2020-06-03T18:55:22.233 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone3, test-zone1} 2020-06-03T18:55:22.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:22.247 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:22.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:22.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:22.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:22.249 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:22.255 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:22.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:22.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:22.256 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:22.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:22.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:22.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:22.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:22.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:22.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:22.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:22.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:22.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:22.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:23.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:23.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:23.240 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone3, test-zone1} 2020-06-03T18:55:23.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-16 zones={test-zone3, test-zone2} 2020-06-03T18:55:23.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:23.257 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:23.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:23.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:23.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:23.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:23.266 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:23.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:23.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:23.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:23.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:23.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:23.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:23.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:23.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:23.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:23.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:23.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:23.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:23.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:24.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:24.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:24.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-16 zones={test-zone3, test-zone2} 2020-06-03T18:55:24.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:24.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:24.267 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:24.311 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:24.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:24.527 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.68.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:24.527 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:24.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:24.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:24.529 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:24.574 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:24.574 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:24.810 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:24.811 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.24.5 target=00000000007.21.9 2020-06-03T18:55:29.812 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:29.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:29.814 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:29.857 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:29.857 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:30.098 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:30.099 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.24.5 target=00000000007.21.9 2020-06-03T18:55:35.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:35.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:35.102 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:35.146 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:35.146 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:35.396 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9","timestamp":"2020-06-03T18:53:51.078731Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:35.397 WARNING:rgw_multi.tests:bucket rhmmtn-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.24.5 target=00000000007.21.9 2020-06-03T18:55:40.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.401 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:40.448 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:40.449 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:40.696 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:40.696 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:40.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.698 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:40.952 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.68.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:40.953 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:40.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:40.955 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:41.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:41.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:41.262 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:41.264 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:41.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.266 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:41.347 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:41.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:41.558 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.68.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.22.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:41.559 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:41.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.561 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:41.608 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:41.608 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:41.966 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.68.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.22.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.24.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:41.967 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-17 2020-06-03T18:55:41.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:41.969 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:42.048 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:42.049 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:42.283 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.50.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.41.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:42.284 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:42.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.286 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:42.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:42.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:42.564 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000004.50.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000011.41.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000009.24.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:42.565 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:42.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.567 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:42.611 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:42.612 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:42.813 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.50.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.41.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:42.814 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:42.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:42.816 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:42.863 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:42.863 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:43.087 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000004.50.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000011.41.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000009.24.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:43.088 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-17 2020-06-03T18:55:43.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.090 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:43.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:43.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:43.342 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.84.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:43.342 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:43.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.344 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:43.426 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:43.426 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:43.661 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000004.84.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000015.40.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:43.661 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:43.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.663 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:43.743 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:43.744 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:43.956 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.84.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:43.957 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:43.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:43.959 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:44.005 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:44.005 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:44.253 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"fAY4LbLpr6tYONDFnfr2MAwhTVZeSa.","ns":""},"count":1},"inc_marker":{"position":"00000000004.84.5","timestamp":"2020-06-03T18:55:16.863950Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"iLrHbL3du87C5yEpZmbK2Rs8b64VmsO","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:16.932440Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"SDIW3nWYFfrMgEy5RgnWcl4sV5yOO5h","ns":""},"count":1},"inc_marker":{"position":"00000000015.40.5","timestamp":"2020-06-03T18:55:17.002974Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:44.254 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-17 2020-06-03T18:55:44.254 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone2, test-zone1} 2020-06-03T18:55:44.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:44.274 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:44.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:44.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:44.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:44.276 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:44.285 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:44.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:44.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:44.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:44.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:44.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:44.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:44.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:44.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:44.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:44.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:44.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:44.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:45.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:45.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:45.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:45.307 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone2, test-zone1} 2020-06-03T18:55:45.307 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone3, test-zone1} 2020-06-03T18:55:45.316 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:45.322 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:45.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:45.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:45.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:45.323 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:45.329 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:45.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:45.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:45.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:45.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:45.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:45.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:45.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:45.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:45.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:45.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:45.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:46.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:46.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:46.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:46.324 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone3, test-zone1} 2020-06-03T18:55:46.325 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-17 zones={test-zone3, test-zone2} 2020-06-03T18:55:46.341 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:46.349 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:46.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:46.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:46.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:46.351 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:46.357 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:46.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:46.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:46.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:46.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:46.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:46.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:46.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:46.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:46.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:46.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:46.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:46.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:47.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:47.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:47.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:47.350 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-17 zones={test-zone3, test-zone2} 2020-06-03T18:55:47.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.352 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:47.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:47.395 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:47.605 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.72.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:47.606 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:47.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.608 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:47.654 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:47.654 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:47.882 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"odOy-VX0HKRMHFxAA0yseey.h30ZTV2","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000008.72.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:47.883 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:47.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:47.885 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:47.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:47.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:48.137 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.72.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:48.137 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:48.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.139 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:48.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:48.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:48.454 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"odOy-VX0HKRMHFxAA0yseey.h30ZTV2","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000008.72.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:48.455 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:48.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.457 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:48.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:48.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:48.748 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.72.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:48.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:48.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:48.750 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:48.795 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:48.795 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:49.072 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.36.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.72.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:49.072 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-18 2020-06-03T18:55:49.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.074 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:49.157 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:49.158 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:49.389 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:49.390 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:49.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.391 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:49.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:49.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:49.658 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000011.40.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000008.54.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:49.658 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:49.659 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.660 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:49.704 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:49.704 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:49.929 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:49.930 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:49.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:49.931 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:49.974 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:50.212 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000004.24.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":2},"inc_marker":{"position":"00000000011.40.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000008.54.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:50.212 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-18 2020-06-03T18:55:50.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.214 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:50.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:50.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:50.511 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.23.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.86.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:50.512 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:50.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.513 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:50.594 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:50.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:50.815 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000017.86.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:50.815 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:50.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:50.817 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:50.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:50.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:51.114 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.23.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.86.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:51.115 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:51.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:51.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:51.116 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:51.404 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"1zS6qq1HZ6ah3LsKF8lTA8JJ0Qqcny7","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-03T18:55:17.070762Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"hNu7iq.tVZhF1qq40Tas7WaSIyccMcr","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5","timestamp":"2020-06-03T18:55:17.140617Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"E1pC0zQX75cNpZDqDPTiVRlM5NcbzEN","ns":""},"count":1},"inc_marker":{"position":"00000000017.86.5","timestamp":"2020-06-03T18:55:17.209947Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:51.404 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-18 2020-06-03T18:55:51.405 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone2, test-zone1} 2020-06-03T18:55:51.417 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:51.425 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:51.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:51.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:51.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:51.432 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:51.442 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:51.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:51.443 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:51.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:51.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:51.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:51.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:51.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:51.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:51.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:51.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:51.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:52.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:52.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:52.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:52.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:52.459 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone2, test-zone1} 2020-06-03T18:55:52.459 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone3, test-zone1} 2020-06-03T18:55:52.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:52.480 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:52.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:52.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:52.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:52.483 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:52.491 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:52.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:52.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:52.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:52.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:52.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:52.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:52.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:52.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:52.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:52.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:52.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:53.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:53.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:53.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:53.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:53.514 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone3, test-zone1} 2020-06-03T18:55:53.515 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-18 zones={test-zone3, test-zone2} 2020-06-03T18:55:53.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:55:53.531 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:53.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:53.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:53.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:53.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:55:53.538 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:55:53.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-03T18:55:53.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-03T18:55:53.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-03T18:55:53.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:53.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:53.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:55:53.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:53.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:53.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-03T18:55:54.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:54.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:54.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-03T18:55:54.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:54.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:54.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-03T18:55:54.576 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-18 zones={test-zone3, test-zone2} 2020-06-03T18:55:54.578 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-06-03T18:55:54.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:54.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:54.623 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:54.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:54.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:54.822 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:55:54.851 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210554.616959_148.1","last_update":"2020-06-03T18:55:54.616959Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:55:54.852 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:55:54.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:54.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:54.853 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:54.889 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:54.890 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:55.120 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210554.616959_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:54.616959Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:55:55.120 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:55:55.121 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:55:55.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.123 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:55.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:55.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:55.376 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210554.616959_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:54.616959Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:55:55.376 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:55:55.377 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:55:55.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.379 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:55.422 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:55.422 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:55.694 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210554.616959_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:54.616959Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:55:55.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:55:55.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.748 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:55.792 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:55.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:55.984 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:55.985 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:55:55.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:55.987 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:56.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:56.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:56.253 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-03T18:55:55.810361Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:56.254 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:55:56.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:56.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:56.256 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:56.298 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:56.299 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:56.511 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:55:56.511 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:55:56.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:56.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:55:56.513 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:55:56.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:55:56.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:55:56.814 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-03T18:55:55.810360Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:55:56.814 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000002.14.10 2020-06-03T18:56:01.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:01.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:01.817 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:01.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:02.091 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:02.092 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:56:02.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:02.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:02.093 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:02.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:02.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:02.384 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:02.384 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:56:02.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:02.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:02.386 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:02.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:02.432 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:02.704 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:02.705 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000002.14.10 2020-06-03T18:56:07.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:07.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:07.708 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:07.754 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:07.754 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:08.025 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:08.025 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:56:08.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.027 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:08.114 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:08.114 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:08.345 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:08.345 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:56:08.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.347 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:08.396 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:08.397 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:08.622 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:08.623 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:56:08.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.625 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:08.668 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:08.669 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:08.908 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:08.909 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:56:08.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:08.911 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:08.954 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:08.954 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:09.181 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:09.182 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:56:09.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:09.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:09.184 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:09.268 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:09.269 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:09.488 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:09.489 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:56:09.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:09.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:09.491 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:09.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:09.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:09.797 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:09.798 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-03T18:56:14.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:14.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:14.801 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:14.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:14.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:15.080 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:15.080 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-03T18:56:20.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:20.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:20.083 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:20.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:20.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:20.370 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:20.371 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-03T18:56:25.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:25.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:25.374 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:25.418 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:25.419 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:25.630 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:25.631 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-03T18:56:30.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:30.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:30.635 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:30.680 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:30.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:30.909 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:30.910 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-03T18:56:35.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:35.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:35.913 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:35.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:35.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:36.180 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:36.181 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:56:36.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:36.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:36.183 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:36.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:36.266 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:36.477 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:36.478 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:56:36.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:36.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:36.480 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:36.772 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:36.772 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:56:36.773 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone2, test-zone1} 2020-06-03T18:56:36.786 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:56:36.835 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:36.835 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:56:36.844 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:36.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:36.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:36.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone2, test-zone1} 2020-06-03T18:56:37.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone3, test-zone1} 2020-06-03T18:56:37.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:56:37.134 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:37.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:56:37.144 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:37.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.357 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone3, test-zone1} 2020-06-03T18:56:37.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone3, test-zone2} 2020-06-03T18:56:37.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:56:37.422 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:37.423 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:56:37.430 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:56:37.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:56:37.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone3, test-zone2} 2020-06-03T18:56:39.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:39.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:39.164 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:39.204 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:39.204 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:39.404 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000033.106.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:56:39.404 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:56:39.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:39.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:39.406 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:39.452 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:39.453 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:39.690 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000009.41.5","timestamp":"2020-06-03T18:56:37.701506Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:39.691 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000033.106.5 target=00000000009.41.5 2020-06-03T18:56:44.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:44.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:44.694 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:44.743 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:44.743 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:44.968 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000009.41.5","timestamp":"2020-06-03T18:56:37.701506Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:44.968 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000033.106.5 target=00000000009.41.5 2020-06-03T18:56:49.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:49.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:49.971 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:50.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:50.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:50.263 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000009.41.5","timestamp":"2020-06-03T18:56:37.701506Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:50.263 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000033.106.5 target=00000000009.41.5 2020-06-03T18:56:55.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:55.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:56:55.267 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:56:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:56:55.313 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:56:55.559 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000009.41.5","timestamp":"2020-06-03T18:56:37.701506Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:56:55.560 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000033.106.5 target=00000000009.41.5 2020-06-03T18:57:00.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:00.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:00.563 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:00.609 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:00.610 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:00.838 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000040.127.5","timestamp":"2020-06-03T18:56:38.917138Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:00.838 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:57:00.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:00.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:00.840 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:00.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:00.882 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:01.100 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000042.131.8"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:01.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:57:01.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.102 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:01.204 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:01.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:01.434 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"lqGu5ZlO93FxH8OQVR3CNtLfa-rcMsN","ns":""},"count":1},"inc_marker":{"position":"00000000042.131.8","timestamp":"2020-06-03T18:56:39.066322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:01.435 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:57:01.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.436 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:01.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:01.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:01.716 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000042.131.8"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:01.717 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:57:01.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:01.718 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:01.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:01.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:02.021 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000033.106.5","timestamp":"2020-06-03T18:56:38.816543Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:02.022 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000042.131.8 target=00000000033.106.5 2020-06-03T18:57:07.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.025 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:07.070 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:07.071 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:07.347 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000042.131.8","timestamp":"2020-06-03T18:56:39.066322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:07.348 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-19 2020-06-03T18:57:07.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.350 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:07.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:07.431 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:07.665 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000048.135.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:07.666 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:57:07.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:07.667 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:07.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:07.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:07.944 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"w-66wYqQa6nrD9h7XBDAilUgcStohF-","ns":""},"count":1},"inc_marker":{"position":"00000000045.128.5","timestamp":"2020-06-03T18:56:38.816543Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:07.944 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000048.135.9 target=00000000045.128.5 2020-06-03T18:57:12.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:12.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:12.947 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:12.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:13.232 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"w-66wYqQa6nrD9h7XBDAilUgcStohF-","ns":""},"count":1},"inc_marker":{"position":"00000000045.128.5","timestamp":"2020-06-03T18:56:38.816543Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:13.233 WARNING:rgw_multi.tests:bucket rhmmtn-19 zone test-zone1 behind zone test-zone2: shard=8 master=00000000048.135.9 target=00000000045.128.5 2020-06-03T18:57:18.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.236 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:18.281 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:18.282 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:18.511 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"w-66wYqQa6nrD9h7XBDAilUgcStohF-","ns":""},"count":1},"inc_marker":{"position":"00000000049.137.9","timestamp":"2020-06-03T18:56:39.015274Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:18.512 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:57:18.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.515 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:18.777 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.137.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:18.778 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:57:18.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:18.780 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:18.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:18.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:19.053 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"w-66wYqQa6nrD9h7XBDAilUgcStohF-","ns":""},"count":1},"inc_marker":{"position":"00000000049.137.9","timestamp":"2020-06-03T18:56:39.015274Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:19.054 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-19 2020-06-03T18:57:19.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.055 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:19.136 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:19.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:19.341 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000053.145.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:19.342 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:57:19.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.343 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:19.425 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:19.426 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:19.646 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"D2X86O6YRUYh.vXu1fbQeZMAo5EyDyH","ns":""},"count":1},"inc_marker":{"position":"00000000053.145.9","timestamp":"2020-06-03T18:56:39.066322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:19.647 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:57:19.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.648 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:19.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:19.727 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:19.939 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000053.145.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:19.940 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:57:19.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:19.946 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:19.996 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:19.996 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:20.248 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"EvgzxTiJYBWvjSZgmZL3ZLcpzhOKlOH","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-03T18:55:55.701537Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"D2X86O6YRUYh.vXu1fbQeZMAo5EyDyH","ns":""},"count":1},"inc_marker":{"position":"00000000053.145.9","timestamp":"2020-06-03T18:56:39.066322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:20.249 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-19 2020-06-03T18:57:20.250 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone2, test-zone1} 2020-06-03T18:57:20.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:20.273 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:20.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.286 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:20.299 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:20.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:20.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:20.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:20.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:20.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:20.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:20.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:20.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:20.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:20.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:21.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:22.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:23.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:24.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:25.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:26.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:27.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:28.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.314 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone2, test-zone1} 2020-06-03T18:57:29.314 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone3, test-zone1} 2020-06-03T18:57:29.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:29.331 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:29.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:29.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:29.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:29.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:29.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:29.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:29.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:29.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:30.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:31.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:31.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:31.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:31.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:31.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:32.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:33.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:34.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:35.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:36.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:37.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:38.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.127 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone3, test-zone1} 2020-06-03T18:57:39.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-19 zones={test-zone3, test-zone2} 2020-06-03T18:57:39.140 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:39.149 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:39.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.172 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:39.185 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-03T18:57:39.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.193 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.194 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.196 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.197 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.198 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.201 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:39.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:39.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:39.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-03T18:57:39.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:39.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:40.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:41.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:42.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:43.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:44.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:45.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:46.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:47.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:48.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:48.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-19 zones={test-zone3, test-zone2} 2020-06-03T18:57:48.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.249 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:48.289 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:48.289 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:48.514 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210646.318601_222.1","last_update":"2020-06-03T18:57:26.318601Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T18:57:48.514 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T18:57:48.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.516 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:48.559 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:48.559 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:48.773 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210646.318601_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:26.318601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:48.774 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T18:57:48.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:48.775 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:48.817 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:48.817 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:49.038 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210646.318601_222.1","last_update":"2020-06-03T18:57:26.318601Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T18:57:49.039 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T18:57:49.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.041 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:49.121 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:49.121 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:49.329 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210646.318601_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:26.318601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:49.329 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T18:57:49.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.331 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:49.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:49.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:49.619 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210646.318601_222.1","last_update":"2020-06-03T18:57:26.318601Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T18:57:49.619 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T18:57:49.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.621 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:49.659 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:49.659 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:49.930 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11569358039035221345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210646.318601_222.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:26.318601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:49.931 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T18:57:49.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:49.933 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:50.008 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:50.009 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:50.243 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210646.337777_204.1","last_update":"2020-06-03T18:57:26.337777Z"},{"marker":"1_1591210580.323556_175.1","last_update":"2020-06-03T18:56:20.323556Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T18:57:50.244 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T18:57:50.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.246 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:50.287 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:50.287 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:50.509 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4441235937469265296},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210646.337777_204.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:26.337777Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.323556_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.323556Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:50.510 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T18:57:50.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.512 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:50.553 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:50.554 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:50.773 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210646.337777_204.1","last_update":"2020-06-03T18:57:26.337777Z"},{"marker":"1_1591210580.323556_175.1","last_update":"2020-06-03T18:56:20.323556Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T18:57:50.774 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T18:57:50.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:50.776 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:50.818 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:50.819 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:51.032 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6026847009502321472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210646.337777_204.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:26.337777Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.323556_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.323556Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:51.033 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T18:57:51.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.035 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:51.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:51.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:51.336 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210624.303685_198.1","last_update":"2020-06-03T18:57:04.303685Z"},{"marker":"1_1591210580.295126_132.1","last_update":"2020-06-03T18:56:20.295126Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.243048_104.1","last_update":"2020-06-03T18:55:17.243048Z"}] 2020-06-03T18:57:51.337 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T18:57:51.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.339 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:51.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:51.421 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:51.633 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210624.303685_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:04.303685Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.295126_132.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.295126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.243048_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.243048Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:51.633 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T18:57:51.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.635 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:51.712 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:51.933 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210624.303685_198.1","last_update":"2020-06-03T18:57:04.303685Z"},{"marker":"1_1591210580.295126_132.1","last_update":"2020-06-03T18:56:20.295126Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591210517.243048_104.1","last_update":"2020-06-03T18:55:17.243048Z"}] 2020-06-03T18:57:51.933 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T18:57:51.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:51.935 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:51.980 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:51.980 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:52.218 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":18141103562818399107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210624.303685_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:04.303685Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.295126_132.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.295126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.243048_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.243048Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T18:57:52.218 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T18:57:52.219 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-06-03T18:57:52.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.254 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:52.290 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:52.290 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:52.472 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:57:52.498 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210672.247152_180.1","last_update":"2020-06-03T18:57:52.247152Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:57:52.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:57:52.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.500 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:52.536 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:52.537 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:52.763 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210672.247152_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:52.247152Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:52.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:57:52.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:57:52.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:52.765 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:52.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:52.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:53.019 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210672.247152_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:52.247152Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:53.020 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:57:53.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:57:53.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.022 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:53.063 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:53.064 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:53.328 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210672.247152_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:52.247152Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:53.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:57:53.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.383 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:53.423 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:53.423 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:53.615 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:53.615 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:53.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.617 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:53.664 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:53.664 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:53.884 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.474359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:53.885 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:53.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:53.887 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:53.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:53.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:54.151 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:54.152 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:54.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.154 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:54.239 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:54.239 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:54.464 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.470373Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:54.464 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:54.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.466 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:54.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:54.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:54.758 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:54.758 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:54.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:54.760 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:54.810 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:54.811 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:55.082 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.336939Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:55.083 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:55.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.085 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:55.129 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:55.130 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:55.362 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:55.362 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:57:55.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.364 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:55.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:55.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:55.639 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:55.640 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:57:55.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.643 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:55.909 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:55.910 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:57:55.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:55.911 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:55.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:55.958 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:56.197 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:56.197 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:57:56.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.199 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:56.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:56.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:56.443 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.27.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:56.444 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:57:56.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.445 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:56.530 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:56.530 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:56.753 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.27.5","timestamp":"2020-06-03T18:57:53.702422Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:56.753 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:57:56.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:56.755 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:56.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:56.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:57.051 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.27.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:57.052 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:57:57.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:57.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:57.054 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:57.098 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:57.098 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:57.347 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.27.5","timestamp":"2020-06-03T18:57:53.702335Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:57.348 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:57:57.348 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:57:57.360 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:57.410 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:57.411 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:57.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:57.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.623 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:57:57.624 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:57:57.632 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:57.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:57.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:57.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:57.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:57.885 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:57:57.885 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:57:57.896 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:57:57.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:57.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:57:57.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:57:58.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:58.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:58.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:57:58.158 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:57:58.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.176 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:58.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:58.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:58.382 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:57:58.417 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210678.170340_188.1","last_update":"2020-06-03T18:57:58.170340Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:57:58.418 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:57:58.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.420 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:58.459 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:58.459 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:58.702 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210678.170340_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:58.170340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:58.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:57:58.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:57:58.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.705 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:58.962 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210678.170340_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:58.170340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:58.962 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:57:58.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:57:58.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:58.965 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:59.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:59.007 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:59.286 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210678.170340_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:57:58.170340Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:57:59.287 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:57:59.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:59.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:59.340 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:59.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:59.380 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:59.597 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:57:59.597 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:57:59.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:59.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:57:59.599 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:57:59.645 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:57:59.646 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:57:59.896 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.474359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:57:59.897 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-03T18:58:04.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:04.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:04.900 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:04.944 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:04.945 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:05.195 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.474359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:05.196 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-03T18:58:10.197 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:10.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:10.198 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:10.244 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:10.244 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:10.494 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-03T18:57:53.474359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:10.495 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-03T18:58:15.496 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:15.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:15.497 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:15.551 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:15.814 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:15.815 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:58:15.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:15.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:15.816 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:15.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:15.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:16.078 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:58:16.078 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:58:16.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.080 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:16.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:16.392 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:16.393 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:58:16.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.395 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:16.670 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:58:16.671 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:58:16.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.673 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:16.720 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:16.720 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:16.980 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:16.981 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:58:16.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:16.983 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:17.264 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.50.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:58:17.264 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:58:17.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:17.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:17.266 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:17.310 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:17.534 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:17.535 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.45.5 2020-06-03T18:58:22.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:22.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:22.539 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:22.583 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:22.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:22.816 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:22.816 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.45.5 2020-06-03T18:58:27.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:27.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:27.820 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:27.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:27.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:28.100 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:28.100 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.45.5 2020-06-03T18:58:33.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:33.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:33.104 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:33.149 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:33.149 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:33.376 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T18:57:53.710399Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:33.376 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.45.5 2020-06-03T18:58:38.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.380 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:38.446 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:38.447 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:38.678 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:38.679 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:58:38.680 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.681 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:38.737 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:38.738 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:38.954 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.52.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:58:38.955 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:58:38.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:38.956 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:39.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:39.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:39.190 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:39.191 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:58:39.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:39.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:39.193 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:39.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:39.414 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.34.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:58:39.414 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:58:39.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:39.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:39.416 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:39.502 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:39.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:39.706 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:39.707 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.34.9 target=00000000003.32.12 2020-06-03T18:58:44.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:44.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:44.710 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:44.992 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:44.993 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.34.9 target=00000000003.32.12 2020-06-03T18:58:49.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:49.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:49.997 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:50.040 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:50.040 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:50.279 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:50.279 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.34.9 target=00000000003.32.12 2020-06-03T18:58:55.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:55.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:58:55.282 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:58:55.327 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:58:55.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:58:55.566 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.32.12","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:58:55.567 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000004.34.9 target=00000000003.32.12 2020-06-03T18:59:00.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:00.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:00.570 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:00.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:00.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:00.849 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.34.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:00.849 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:00.850 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:00.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:00.851 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:00.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:00.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:01.153 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.34.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:01.154 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:01.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:01.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:01.155 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:01.200 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:01.201 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:01.463 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.34.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:01.464 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:01.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:59:01.481 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:01.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.532 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:01.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:01.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:59:01.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:59:01.913 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:01.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.967 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:01.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:01.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:02.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:59:02.366 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:59:02.378 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:02.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:02.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:02.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:02.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:02.440 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:02.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:02.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:59:02.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:02.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:02.819 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:02.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:02.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:03.036 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:59:03.070 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210742.813132_206.1","last_update":"2020-06-03T18:59:02.813132Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:59:03.070 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:59:03.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.072 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:03.115 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:03.115 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:03.358 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210742.813132_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:02.813132Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:03.359 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:59:03.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:59:03.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.361 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:03.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:03.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:03.621 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210742.813132_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:02.813132Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:03.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:59:03.623 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:59:03.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:03.625 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:03.674 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:03.949 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210742.813132_206.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:02.813132Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:03.950 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:59:04.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.004 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:04.042 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:04.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:04.323 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:04.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:04.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.325 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:04.385 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:04.385 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:04.632 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:04.633 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:04.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.635 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:04.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:04.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:04.896 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:04.897 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:04.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:04.899 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:04.982 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:04.983 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:05.224 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:05.225 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:05.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.227 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:05.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:05.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:05.527 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:05.528 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:05.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.531 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:05.578 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:05.578 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:05.851 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:05.851 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-20 2020-06-03T18:59:05.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:05.853 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:05.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:06.170 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000006.58.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:06.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:59:06.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:06.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:06.173 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:06.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:06.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:06.462 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:06.463 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.58.17 target=00000000004.52.9 2020-06-03T18:59:11.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:11.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:11.467 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:11.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:11.750 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:11.751 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.58.17 target=00000000004.52.9 2020-06-03T18:59:16.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:16.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:16.754 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:16.797 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:17.033 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-03T18:57:59.294022Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:17.034 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.58.17 target=00000000004.52.9 2020-06-03T18:59:22.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.037 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:22.084 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:22.330 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.60.14","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:22.331 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:59:22.332 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.333 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:22.375 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:22.624 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.60.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:22.624 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:59:22.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.627 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:22.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:22.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:22.903 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.60.14","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:22.904 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-20 2020-06-03T18:59:22.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:22.907 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:22.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:22.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:23.170 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.42.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:23.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:23.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:23.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:23.172 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:23.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:23.256 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:23.495 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.40.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:23.496 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.42.14 target=00000000006.40.17 2020-06-03T18:59:28.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:28.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:28.499 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:28.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:28.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:28.765 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.40.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:28.766 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.42.14 target=00000000006.40.17 2020-06-03T18:59:33.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:33.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:33.770 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:33.812 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:33.813 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:34.051 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.40.17","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:34.052 WARNING:rgw_multi.tests:bucket rhmmtn-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.42.14 target=00000000006.40.17 2020-06-03T18:59:39.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.055 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:39.098 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:39.099 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:39.330 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.42.14","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:39.331 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:39.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.332 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:39.371 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:39.371 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:39.569 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.42.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:39.569 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:39.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:39.571 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:39.622 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:39.622 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:39.860 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.42.14","timestamp":"2020-06-03T18:59:03.958576Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:39.861 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-20 2020-06-03T18:59:39.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:59:39.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:39.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:39.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:39.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:39.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:39.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:39.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.275 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone2, test-zone1} 2020-06-03T18:59:40.276 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:59:40.284 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:40.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:40.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:40.842 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone1} 2020-06-03T18:59:40.843 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:59:40.861 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:40.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.918 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:40.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:40.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:41.290 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-20 zones={test-zone3, test-zone2} 2020-06-03T18:59:41.291 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-06-03T18:59:41.292 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-21 2020-06-03T18:59:41.322 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-22 2020-06-03T18:59:41.375 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-23 2020-06-03T18:59:41.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.477 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:41.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:41.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:41.686 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:59:41.719 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210781.464609_236.1","last_update":"2020-06-03T18:59:41.464609Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:59:41.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:59:41.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.722 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:41.759 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:41.759 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:41.981 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210781.464609_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:41.464609Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:41.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:59:41.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:59:41.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:41.984 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:42.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:42.027 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:42.241 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210781.464609_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:41.464609Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:42.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:59:42.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:59:42.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:42.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:42.244 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:42.286 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:42.286 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:42.565 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210781.464609_236.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:41.464609Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:42.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:59:42.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:42.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:42.783 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:42.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:43.009 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:43.010 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:43.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.011 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:43.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:43.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:43.268 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000003.17.11","timestamp":"2020-06-03T18:59:42.734240Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:43.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:43.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.270 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:43.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:43.531 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:43.531 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:43.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.533 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:43.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:43.854 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000003.17.11","timestamp":"2020-06-03T18:59:42.738305Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:43.855 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:43.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:43.856 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:43.939 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:43.940 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:44.169 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:44.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:44.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.172 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:44.217 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:44.217 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:44.491 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.17.11","timestamp":"2020-06-03T18:59:42.621827Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:44.492 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-21 2020-06-03T18:59:44.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.493 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:44.576 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:44.577 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:44.808 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:44.809 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-21 2020-06-03T18:59:44.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:44.810 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:44.855 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:44.855 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:45.097 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000002.24.5","timestamp":"2020-06-03T18:59:43.018319Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:45.098 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-21 2020-06-03T18:59:45.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.100 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:45.141 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:45.141 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:45.384 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:45.384 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-21 2020-06-03T18:59:45.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.386 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:45.429 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:45.430 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:45.669 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000002.24.5","timestamp":"2020-06-03T18:59:43.014334Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:45.669 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-21 2020-06-03T18:59:45.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.671 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:45.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:45.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:45.970 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:45.970 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-21 2020-06-03T18:59:45.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:45.972 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:46.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:46.054 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:46.296 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000002.21.5","timestamp":"2020-06-03T18:59:42.914697Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:46.297 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-21 2020-06-03T18:59:46.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:46.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:46.298 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:46.597 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:46.597 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-21 2020-06-03T18:59:46.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:46.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:46.599 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:46.644 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:46.644 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:46.882 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"HCGz6VT2lnRSBTsQyx3-4zK9MIWYXRr","ns":""},"count":2},"inc_marker":{"position":"00000000002.21.5","timestamp":"2020-06-03T18:59:42.919637Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:46.883 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-21 2020-06-03T18:59:46.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-21 zones={test-zone2, test-zone1} 2020-06-03T18:59:46.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:46.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:46.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:46.947 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:46.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:46.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.107 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-21 zones={test-zone2, test-zone1} 2020-06-03T18:59:47.108 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-21 zones={test-zone3, test-zone1} 2020-06-03T18:59:47.117 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:47.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:47.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.322 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-21 zones={test-zone3, test-zone1} 2020-06-03T18:59:47.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-21 zones={test-zone3, test-zone2} 2020-06-03T18:59:47.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:47.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:47.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:47.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:47.537 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-21 zones={test-zone3, test-zone2} 2020-06-03T18:59:47.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:47.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:47.539 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:47.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:47.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:47.799 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.143.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:47.800 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:47.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:47.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:47.801 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:47.848 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:47.848 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:48.108 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000002.143.5","timestamp":"2020-06-03T18:59:42.974284Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:48.108 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:48.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.110 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:48.152 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:48.152 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:48.372 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.143.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:48.373 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:48.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.375 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:48.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:48.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:48.690 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000002.143.5","timestamp":"2020-06-03T18:59:42.974276Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:48.690 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:48.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:48.692 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:48.776 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:49.000 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.143.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:49.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:49.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.003 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:49.048 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:49.048 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:49.322 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.143.5","timestamp":"2020-06-03T18:59:42.645610Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:49.323 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-22 2020-06-03T18:59:49.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.325 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:49.374 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:49.374 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:49.610 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.151.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:49.610 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-22 2020-06-03T18:59:49.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.612 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:49.655 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:49.655 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:49.888 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000003.151.11","timestamp":"2020-06-03T18:59:42.766394Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:49.889 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-22 2020-06-03T18:59:49.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:49.890 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:49.933 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:49.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:50.172 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.151.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:50.173 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-22 2020-06-03T18:59:50.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.175 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:50.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:50.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:50.468 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000003.151.11","timestamp":"2020-06-03T18:59:42.766388Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:50.469 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-22 2020-06-03T18:59:50.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.470 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:50.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:50.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:50.706 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.157.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:50.706 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-22 2020-06-03T18:59:50.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:50.708 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:50.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:50.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:51.031 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000002.157.5","timestamp":"2020-06-03T18:59:42.914970Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:51.031 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-22 2020-06-03T18:59:51.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:51.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:51.033 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:51.332 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.157.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:51.332 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-22 2020-06-03T18:59:51.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:51.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:51.334 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:51.380 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:51.381 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:51.629 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Gza6JZo3BOYMPOO9.S29QROPeJZQEP.","ns":""},"count":1},"inc_marker":{"position":"00000000002.157.5","timestamp":"2020-06-03T18:59:42.920243Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:51.630 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-22 2020-06-03T18:59:51.630 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-22 zones={test-zone2, test-zone1} 2020-06-03T18:59:51.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:51.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.695 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:51.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:51.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:51.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:51.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-22 zones={test-zone2, test-zone1} 2020-06-03T18:59:51.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-22 zones={test-zone3, test-zone1} 2020-06-03T18:59:51.863 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:51.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:51.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:51.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:51.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:51.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:52.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-22 zones={test-zone3, test-zone1} 2020-06-03T18:59:52.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-22 zones={test-zone3, test-zone2} 2020-06-03T18:59:52.076 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:52.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:52.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:52.128 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:52.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:52.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:52.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:52.291 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-22 zones={test-zone3, test-zone2} 2020-06-03T18:59:52.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.293 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:52.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:52.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:52.553 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:52.553 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:52.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.555 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:52.600 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:52.601 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:52.837 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-03T18:59:42.716417Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:52.837 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:52.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:52.839 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:52.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:52.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:53.104 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:53.105 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:53.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.107 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:53.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:53.191 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:53.427 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-03T18:59:42.716417Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:53.427 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:53.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.429 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:53.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:53.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:53.715 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.19.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:53.716 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:53.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:53.718 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:53.762 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:53.763 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:54.034 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-03T18:59:42.716417Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:54.035 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-23 2020-06-03T18:59:54.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.036 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:54.078 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:54.079 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:54.317 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:54.318 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-23 2020-06-03T18:59:54.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.319 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:54.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:54.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:54.613 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000002.28.5","timestamp":"2020-06-03T18:59:43.170299Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:54.613 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-23 2020-06-03T18:59:54.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.615 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:54.663 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:54.663 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:54.898 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:54.898 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-23 2020-06-03T18:59:54.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:54.900 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:54.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:54.944 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:55.191 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000002.28.5","timestamp":"2020-06-03T18:59:43.166306Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:55.192 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-23 2020-06-03T18:59:55.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.194 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:55.275 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:55.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:55.494 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.30.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:55.494 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-23 2020-06-03T18:59:55.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.496 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:55.570 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:55.570 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:55.813 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.11","timestamp":"2020-06-03T18:59:42.958318Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:55.814 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-23 2020-06-03T18:59:55.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:55.816 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:55.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:55.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:56.113 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.30.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:56.113 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-23 2020-06-03T18:59:56.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:56.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:56.116 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:56.160 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:56.160 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:56.415 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"ariHjOpOUrvlHrXFVh6we8NbwKBCufY","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.11","timestamp":"2020-06-03T18:59:42.962322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:56.415 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-23 2020-06-03T18:59:56.416 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-23 zones={test-zone2, test-zone1} 2020-06-03T18:59:56.428 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:56.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:56.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.647 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-23 zones={test-zone2, test-zone1} 2020-06-03T18:59:56.648 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-23 zones={test-zone3, test-zone1} 2020-06-03T18:59:56.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:56.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.707 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:56.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:56.867 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-23 zones={test-zone3, test-zone1} 2020-06-03T18:59:56.868 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-23 zones={test-zone3, test-zone2} 2020-06-03T18:59:56.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T18:59:56.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T18:59:56.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T18:59:56.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:57.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:57.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T18:59:57.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-23 zones={test-zone3, test-zone2} 2020-06-03T18:59:57.095 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-06-03T18:59:57.096 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-24 2020-06-03T18:59:57.129 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-25 2020-06-03T18:59:57.183 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-26 2020-06-03T18:59:57.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.355 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:57.390 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:57.390 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:57.562 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T18:59:57.589 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210797.348893_276.1","last_update":"2020-06-03T18:59:57.348893Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T18:59:57.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T18:59:57.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.591 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:57.846 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210797.348893_276.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:57.348893Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:57.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T18:59:57.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T18:59:57.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:57.849 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:57.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:57.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:58.108 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210797.348893_276.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:57.348893Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:58.109 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T18:59:58.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T18:59:58.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:58.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:58.111 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:58.152 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:58.152 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:58.528 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210797.348893_276.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:59:57.348893Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T18:59:58.529 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T18:59:58.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:58.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:58.786 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:58.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:58.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:59.009 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.11.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:59.010 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T18:59:59.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.011 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:59.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:59.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:59.284 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.6","timestamp":"2020-06-03T18:59:58.742310Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:59.285 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T18:59:59.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.289 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:59.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:59.334 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:59.546 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.11.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T18:59:59.547 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T18:59:59.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.549 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:59.632 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:59.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T18:59:59.870 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.6","timestamp":"2020-06-03T18:59:58.746333Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T18:59:59.871 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T18:59:59.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T18:59:59.873 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T18:59:59.949 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T18:59:59.950 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.11.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:00.161 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T19:00:00.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.163 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:00.208 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:00.208 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:00.482 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.11.6","timestamp":"2020-06-03T18:59:58.538283Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:00.482 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-24 2020-06-03T19:00:00.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.484 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:00.566 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:00.566 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:00.798 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.10.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:00.799 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-24 2020-06-03T19:00:00.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:00.801 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:00.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:00.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:01.089 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.12","timestamp":"2020-06-03T18:59:58.977854Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:01.090 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-24 2020-06-03T19:00:01.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.092 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:01.382 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.10.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:01.382 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-24 2020-06-03T19:00:01.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.385 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:01.428 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:01.428 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:01.668 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.12","timestamp":"2020-06-03T18:59:59.018398Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:01.669 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-24 2020-06-03T19:00:01.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.672 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:01.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:01.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:01.972 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.8.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:01.973 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-24 2020-06-03T19:00:01.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:01.994 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:02.059 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:02.296 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.12","timestamp":"2020-06-03T18:59:58.847697Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:02.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-24 2020-06-03T19:00:02.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:02.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:02.298 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:02.338 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:02.339 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:02.671 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.8.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:02.672 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-24 2020-06-03T19:00:02.672 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:02.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:02.673 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:02.722 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:02.722 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:02.950 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.12","timestamp":"2020-06-03T18:59:58.894324Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:02.950 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-24 2020-06-03T19:00:02.951 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-24 zones={test-zone2, test-zone1} 2020-06-03T19:00:02.960 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:03.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.011 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:03.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:03.088 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-24 zones={test-zone2, test-zone1} 2020-06-03T19:00:03.088 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-24 zones={test-zone3, test-zone1} 2020-06-03T19:00:03.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:03.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.147 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:03.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:03.219 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-24 zones={test-zone3, test-zone1} 2020-06-03T19:00:03.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-24 zones={test-zone3, test-zone2} 2020-06-03T19:00:03.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:03.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.279 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:03.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:03.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:03.351 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-24 zones={test-zone3, test-zone2} 2020-06-03T19:00:03.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.353 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:03.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:03.627 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.48.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:03.627 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:03.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.629 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:03.911 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.12","timestamp":"2020-06-03T18:59:58.990309Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:03.912 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:03.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:03.913 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:03.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:03.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:04.172 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.48.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:04.173 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:04.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.174 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:04.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:04.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:04.488 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.12","timestamp":"2020-06-03T18:59:58.982276Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:04.489 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:04.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.491 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:04.580 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:04.580 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:04.790 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.48.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:04.790 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:04.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:04.792 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:05.106 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.12","timestamp":"2020-06-03T18:59:58.677088Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:05.106 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-25 2020-06-03T19:00:05.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.108 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:05.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:05.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:05.420 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.40.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:05.421 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-25 2020-06-03T19:00:05.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.422 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:05.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:05.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:05.703 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.40.6","timestamp":"2020-06-03T18:59:58.774367Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:05.704 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-25 2020-06-03T19:00:05.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.705 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:05.746 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:05.746 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:05.978 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.40.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:05.979 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-25 2020-06-03T19:00:05.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:05.980 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:06.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:06.027 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:06.259 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.40.6","timestamp":"2020-06-03T18:59:58.774357Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:06.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-25 2020-06-03T19:00:06.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.261 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:06.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:06.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:06.531 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.35.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:06.531 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-25 2020-06-03T19:00:06.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.533 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:06.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:06.751 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:06.860 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.12","timestamp":"2020-06-03T18:59:58.849064Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:06.861 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-25 2020-06-03T19:00:06.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:06.862 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:06.945 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:06.945 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:07.169 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.35.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:07.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-25 2020-06-03T19:00:07.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:07.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:07.172 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:07.216 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:07.217 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:07.489 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.12","timestamp":"2020-06-03T18:59:58.894352Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:07.489 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-25 2020-06-03T19:00:07.490 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-25 zones={test-zone2, test-zone1} 2020-06-03T19:00:07.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:07.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:07.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:07.624 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-25 zones={test-zone2, test-zone1} 2020-06-03T19:00:07.624 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-25 zones={test-zone3, test-zone1} 2020-06-03T19:00:07.635 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:07.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:07.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:07.760 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-25 zones={test-zone3, test-zone1} 2020-06-03T19:00:07.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-25 zones={test-zone3, test-zone2} 2020-06-03T19:00:07.771 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:07.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:07.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:07.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:07.895 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-25 zones={test-zone3, test-zone2} 2020-06-03T19:00:07.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:07.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:07.897 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:07.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:08.159 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:08.159 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:08.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.161 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:08.207 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:08.207 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:08.459 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-03T18:59:59.146302Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:08.460 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:08.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.462 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:08.501 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:08.502 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:08.719 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:08.720 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:08.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:08.722 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:08.803 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:08.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:09.032 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-03T18:59:59.182386Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:09.112 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:09.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.114 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:09.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:09.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:09.385 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:09.385 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:09.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.387 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:09.432 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:09.433 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:09.708 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-03T18:59:58.763835Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:09.709 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-26 2020-06-03T19:00:09.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.711 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:09.992 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:09.992 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-26 2020-06-03T19:00:09.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:09.994 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:10.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:10.284 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.12","timestamp":"2020-06-03T18:59:59.018353Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:10.285 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-26 2020-06-03T19:00:10.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.287 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:10.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:10.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:10.568 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:10.568 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-26 2020-06-03T19:00:10.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.570 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:10.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:10.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:10.847 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.18.12","timestamp":"2020-06-03T18:59:58.977489Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:10.848 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-26 2020-06-03T19:00:10.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:10.850 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:10.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:10.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:11.105 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:11.106 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-26 2020-06-03T19:00:11.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.108 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:11.214 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:11.406 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:11.459 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.6","timestamp":"2020-06-03T18:59:58.890275Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:11.460 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-26 2020-06-03T19:00:11.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.462 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:11.544 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:11.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:11.765 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:00:11.765 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-26 2020-06-03T19:00:11.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:11.767 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:11.813 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:11.813 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:12.062 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.6","timestamp":"2020-06-03T18:59:58.711187Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:00:12.063 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-26 2020-06-03T19:00:12.064 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-26 zones={test-zone2, test-zone1} 2020-06-03T19:00:12.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:12.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.123 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:12.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:12.195 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-26 zones={test-zone2, test-zone1} 2020-06-03T19:00:12.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-26 zones={test-zone3, test-zone1} 2020-06-03T19:00:12.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:12.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:12.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:12.326 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-26 zones={test-zone3, test-zone1} 2020-06-03T19:00:12.327 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-26 zones={test-zone3, test-zone2} 2020-06-03T19:00:12.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:00:12.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:00:12.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-03T19:00:12.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-03T19:00:12.476 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-26 zones={test-zone3, test-zone2} 2020-06-03T19:00:12.477 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-06-03T19:00:12.478 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-27 2020-06-03T19:00:12.511 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-28 2020-06-03T19:00:12.570 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-29 2020-06-03T19:00:12.806 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-06-03T19:00:12.807 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-30 2020-06-03T19:00:12.845 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-31 2020-06-03T19:00:12.921 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-32 2020-06-03T19:00:13.158 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-06-03T19:00:13.159 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-33 2020-06-03T19:00:13.189 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-34 2020-06-03T19:00:13.245 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-35 2020-06-03T19:00:15.502 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-06-03T19:00:15.503 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-36 2020-06-03T19:00:15.544 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-37 2020-06-03T19:00:15.595 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-38 2020-06-03T19:00:15.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:15.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:15.646 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:15.684 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:15.855 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T19:00:15.893 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210815.625665_358.1","last_update":"2020-06-03T19:00:15.625665Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:00:15.894 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:00:15.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:15.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:15.896 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:15.938 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:15.939 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:16.204 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210815.625665_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:15.625665Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:16.204 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:00:16.205 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:00:16.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:16.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:16.206 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:16.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:16.248 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:16.479 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210815.625665_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:15.625665Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:16.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:00:16.480 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:00:16.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:16.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:16.482 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:16.801 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210815.625665_358.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:15.625665Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:16.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:00:17.175 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-06-03T19:00:17.176 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-39 2020-06-03T19:00:17.211 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-40 2020-06-03T19:00:17.266 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-41 2020-06-03T19:00:17.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.323 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:17.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:17.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 2020-06-03T19:00:17.537 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210282.699710_124.1","last_update":"2020-06-03T18:51:22.699710Z"},{"marker":"1_1591210817.301177_374.1","last_update":"2020-06-03T19:00:17.301177Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:00:17.537 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:00:17.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.539 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:17.574 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:17.575 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:17.795 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210817.301177_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.301177Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:17.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:00:17.796 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:00:17.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:17.797 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:18.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210817.301177_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.301177Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:18.057 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:00:18.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:00:18.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:18.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:00:18.059 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:00:18.100 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:18.100 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:18.375 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210817.301177_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.301177Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:00:18.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:00:18.376 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-06-03T19:00:18.377 INFO:teuthology.orchestra.run:waiting for 300 2020-06-03T19:00:18.378 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:00:18.373+0000 7fd240aef700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18473) UID: 0 2020-06-03T19:00:18.378 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:00:18.373+0000 7fd2581450c0 -1 shutting down 2020-06-03T19:00:23.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-06-03T19:00:24.479 INFO:tasks.rgw.c1.client.1:Stopped 2020-06-03T19:00:24.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:00:24.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:00:24.481 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:00:24.520 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:24.520 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:24.550 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:00:24.543+0000 7f9cc792c0c0 0 NOTICE: overriding master zone: 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:00:24.550 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:00:24.543+0000 7f9cc792c0c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:00:24.550 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:00:24.543+0000 7f9cc792c0c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:00:24.552 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:00:24.547+0000 7f9cc792c0c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:00:24.552 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:00:24.547+0000 7f9cc792c0c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:00:24.566 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:00:24.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:00:24.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:00:24.568 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:00:24.650 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:00:24.651 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:00:24.960 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"c667fc06-9ec1-4c12-8825-c7c36661d280","epoch":1,"predecessor_uuid":"f641ddd2-34e8-45d4-a5d7-f7dfb77827d9","sync_status":["1_1591210282.699710_124.1","1_1591210817.301177_374.1","",""],"period_map":{"id":"c667fc06-9ec1-4c12-8825-c7c36661d280","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"3ee2667c-4516-4846-aa22-24f7de69bd8e","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"3ee2667c-4516-4846-aa22-24f7de69bd8e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":3} 2020-06-03T19:00:24.961 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-06-03T19:00:29.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-06-03T19:01:24.963 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-42 2020-06-03T19:01:25.037 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-43 2020-06-03T19:01:25.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:01:25.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:01:25.072 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:01:25.110 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:01:25.110 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:01:25.307 INFO:teuthology.orchestra.run.smithi061.stderr:No --period given, using current period=c667fc06-9ec1-4c12-8825-c7c36661d280 2020-06-03T19:01:25.345 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591210885.066318_218.1","last_update":"2020-06-03T19:01:25.066318Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:01:25.345 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-06-03T19:01:25.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:01:25.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:01:25.347 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:01:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:01:25.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:01:25.611 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c667fc06-9ec1-4c12-8825-c7c36661d280","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"1_1591210885.066318_218.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:01:25.066318Z","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2020-06-03T19:01:25.616 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-06-03T19:01:25.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:01:25.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:01:25.617 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:01:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:01:25.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:01:25.722 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:01:25.719+0000 7f76a87940c0 0 NOTICE: overriding master zone: 3ee2667c-4516-4846-aa22-24f7de69bd8e 2020-06-03T19:01:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:01:25.719+0000 7f76a87940c0 0 failed reading obj info from .rgw.root:zone_info.3ee2667c-4516-4846-aa22-24f7de69bd8e: (2) No such file or directory 2020-06-03T19:01:25.724 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:01:25.719+0000 7f76a87940c0 0 WARNING: could not read zone params for zone id=3ee2667c-4516-4846-aa22-24f7de69bd8e name=test-zone2 2020-06-03T19:01:25.725 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:01:25.719+0000 7f76a87940c0 0 failed reading obj info from .rgw.root:zone_info.b9904e04-437e-4cce-bbd1-9073303d0345: (2) No such file or directory 2020-06-03T19:01:25.726 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:01:25.719+0000 7f76a87940c0 0 WARNING: could not read zone params for zone id=b9904e04-437e-4cce-bbd1-9073303d0345 name=test-zone4 2020-06-03T19:01:25.738 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:01:25.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:01:25.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:01:25.741 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:01:25.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:01:25.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:01:26.118 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":1,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:01:26.119 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-06-03T19:02:26.120 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-44 2020-06-03T19:02:26.168 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-45 2020-06-03T19:02:26.253 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-03T19:02:26.254 INFO:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-03T19:02:26.258 INFO:tasks.rgw.c1.client.1:Started 2020-06-03T19:02:26.259 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2020-06-03T19:02:26.273 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T19:02:26.274 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T19:02:26.278 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2020-06-03T19:02:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-03T19:02:27.280 INFO:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2020-06-03T19:02:27.297 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-03T19:02:27.298 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2020-06-03T19:02:27.302 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-03T19:02:27.303 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2020-06-03T19:02:27.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.305 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:27.382 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:27.383 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:27.592 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:02:27.624 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591210946.220397_140.1","last_update":"2020-06-03T19:02:26.220397Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:02:27.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:02:27.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.626 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:27.665 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:27.665 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:27.886 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210946.220397_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:26.220397Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:02:27.886 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:02:27.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:02:27.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:27.888 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:27.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:27.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:28.153 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210946.220397_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:26.220397Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:02:28.154 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:02:28.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:02:28.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:28.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:28.156 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:28.196 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:28.196 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:28.480 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210946.220397_140.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:26.220397Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:02:28.480 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:02:28.481 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone2, test-zone1} 2020-06-03T19:02:28.503 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:28.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:28.621 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone2, test-zone1} 2020-06-03T19:02:28.622 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone3, test-zone1} 2020-06-03T19:02:28.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:28.694 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:28.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone3, test-zone1} 2020-06-03T19:02:28.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone1, test-zone2} 2020-06-03T19:02:28.775 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:28.822 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:28.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone1, test-zone2} 2020-06-03T19:02:28.894 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone3, test-zone2} 2020-06-03T19:02:28.906 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:28.958 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.025 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone3, test-zone2} 2020-06-03T19:02:29.026 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone1, test-zone3} 2020-06-03T19:02:29.038 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.161 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone1, test-zone3} 2020-06-03T19:02:29.162 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-39 zones={test-zone2, test-zone3} 2020-06-03T19:02:29.174 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-39 zones={test-zone2, test-zone3} 2020-06-03T19:02:29.294 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone2, test-zone1} 2020-06-03T19:02:29.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.428 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone2, test-zone1} 2020-06-03T19:02:29.428 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone3, test-zone1} 2020-06-03T19:02:29.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.490 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.558 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone3, test-zone1} 2020-06-03T19:02:29.559 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone1, test-zone2} 2020-06-03T19:02:29.570 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.622 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone1, test-zone2} 2020-06-03T19:02:29.691 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone3, test-zone2} 2020-06-03T19:02:29.702 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.750 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone3, test-zone2} 2020-06-03T19:02:29.818 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone1, test-zone3} 2020-06-03T19:02:29.830 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:29.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:29.948 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone1, test-zone3} 2020-06-03T19:02:29.949 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-40 zones={test-zone2, test-zone3} 2020-06-03T19:02:29.960 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-40 zones={test-zone2, test-zone3} 2020-06-03T19:02:30.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone2, test-zone1} 2020-06-03T19:02:30.099 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.146 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.219 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone2, test-zone1} 2020-06-03T19:02:30.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone3, test-zone1} 2020-06-03T19:02:30.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.278 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone3, test-zone1} 2020-06-03T19:02:30.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone1, test-zone2} 2020-06-03T19:02:30.358 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.476 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone1, test-zone2} 2020-06-03T19:02:30.477 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone3, test-zone2} 2020-06-03T19:02:30.490 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.538 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone3, test-zone2} 2020-06-03T19:02:30.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone1, test-zone3} 2020-06-03T19:02:30.616 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.662 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone1, test-zone3} 2020-06-03T19:02:30.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-41 zones={test-zone2, test-zone3} 2020-06-03T19:02:30.750 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.798 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:30.871 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-41 zones={test-zone2, test-zone3} 2020-06-03T19:02:30.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone2, test-zone1} 2020-06-03T19:02:30.884 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:30.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone2, test-zone1} 2020-06-03T19:02:31.004 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone3, test-zone1} 2020-06-03T19:02:31.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.062 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone3, test-zone1} 2020-06-03T19:02:31.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone1, test-zone2} 2020-06-03T19:02:31.144 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.194 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.266 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone1, test-zone2} 2020-06-03T19:02:31.267 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone3, test-zone2} 2020-06-03T19:02:31.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.334 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.402 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone3, test-zone2} 2020-06-03T19:02:31.402 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone1, test-zone3} 2020-06-03T19:02:31.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.534 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone1, test-zone3} 2020-06-03T19:02:31.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-42 zones={test-zone2, test-zone3} 2020-06-03T19:02:31.547 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.594 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.670 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-42 zones={test-zone2, test-zone3} 2020-06-03T19:02:31.671 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone2, test-zone1} 2020-06-03T19:02:31.682 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.730 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.803 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone2, test-zone1} 2020-06-03T19:02:31.804 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone3, test-zone1} 2020-06-03T19:02:31.814 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.862 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:31.932 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone3, test-zone1} 2020-06-03T19:02:31.933 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone1, test-zone2} 2020-06-03T19:02:31.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:31.994 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone1, test-zone2} 2020-06-03T19:02:32.063 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone3, test-zone2} 2020-06-03T19:02:32.074 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.126 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone3, test-zone2} 2020-06-03T19:02:32.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone1, test-zone3} 2020-06-03T19:02:32.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.258 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.328 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone1, test-zone3} 2020-06-03T19:02:32.328 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-43 zones={test-zone2, test-zone3} 2020-06-03T19:02:32.340 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-43 zones={test-zone2, test-zone3} 2020-06-03T19:02:32.465 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone2, test-zone1} 2020-06-03T19:02:32.477 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.591 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone2, test-zone1} 2020-06-03T19:02:32.591 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone3, test-zone1} 2020-06-03T19:02:32.601 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone3, test-zone1} 2020-06-03T19:02:32.724 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone1, test-zone2} 2020-06-03T19:02:32.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone1, test-zone2} 2020-06-03T19:02:32.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone3, test-zone2} 2020-06-03T19:02:32.865 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:32.914 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:32.986 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone3, test-zone2} 2020-06-03T19:02:32.987 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone1, test-zone3} 2020-06-03T19:02:33.002 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.120 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone1, test-zone3} 2020-06-03T19:02:33.121 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-44 zones={test-zone2, test-zone3} 2020-06-03T19:02:33.139 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.264 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-44 zones={test-zone2, test-zone3} 2020-06-03T19:02:33.265 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone2, test-zone1} 2020-06-03T19:02:33.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.391 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone2, test-zone1} 2020-06-03T19:02:33.392 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone3, test-zone1} 2020-06-03T19:02:33.400 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone3, test-zone1} 2020-06-03T19:02:33.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone1, test-zone2} 2020-06-03T19:02:33.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.582 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.645 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone1, test-zone2} 2020-06-03T19:02:33.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone3, test-zone2} 2020-06-03T19:02:33.656 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.706 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.771 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone3, test-zone2} 2020-06-03T19:02:33.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone1, test-zone3} 2020-06-03T19:02:33.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.834 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:33.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone1, test-zone3} 2020-06-03T19:02:33.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-45 zones={test-zone2, test-zone3} 2020-06-03T19:02:33.917 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:02:33.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:02:34.038 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-45 zones={test-zone2, test-zone3} 2020-06-03T19:02:34.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.039 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:34.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:34.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:34.327 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210125.646195_38.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.646195632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"write"}}},{"id":"1_1591210125.653449_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.653449077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.551191_41.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.551191174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.561114_44.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.561114974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.562915_45.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.562915114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.566240_46.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.566240457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.586275_47.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.586275825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.590554_48.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.590554050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.592599_49.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.592599034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.597559_50.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.597559865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.805687_51.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.805687792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.809012_52.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.809012918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.810514_53.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.810514753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.813548_54.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.813548628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.228196_55.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.228196711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.232832_57.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.232832116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.234688_58.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.234688598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.238737_59.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.238737032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.257320_60.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.257320059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.262348_61.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.262348221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.264759_62.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.264759978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.269414_63.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.269414349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.308125_64.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.308125120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.312149_65.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.312149093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.313967_66.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.313967428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.317549_67.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.317549729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.000602_68.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.000602707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.004506_70.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.004506898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.006174_71.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.006174782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.009372_72.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.009372170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.031860_73.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.031860332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.035738_74.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.035738385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.037460_75.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.037460166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.040767_76.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.040767522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.066084_77.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.066084318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.070207_78.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.070207165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.072017_79.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.072017297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.075542_80.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.075542707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.103016_81.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.103016563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.107358_82.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.107358242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.109115_83.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.109115163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.112561_84.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.112561397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.165315_85.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.165315641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.169519_86.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.169519322Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.171422_87.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.171422660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.174741_88.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.174741498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.266352_89.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.266352000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.270156_90.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.270156689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.271931_91.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.271931107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.275233_92.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.275233693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.369098_93.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.369098464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.373277_94.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.373277856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.375072_95.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.375072710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.378378_96.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.378378667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.421365_97.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.421365255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.424919_98.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.424919143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.426516_99.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.426516831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.429676_100.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.429676437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.519909_101.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.519909980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.526123_102.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.526123681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.528147_103.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.528147872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.531714_104.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.531714552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210276.048639_105.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.048639896Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.053020_107.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.053020734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.054976_108.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.054976082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.058926_109.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.058926026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.082565_110.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.082565090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.087382_111.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.087382132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.089688_112.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.089688894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.093302_113.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.093302341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.133481_114.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.133481597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.138334_115.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.138334193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.140074_116.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.140074226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.144087_117.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.144087557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210282.550024_118.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.550024390Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.556399_120.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.556399419Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.613019_121.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.613019625Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.620319_122.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.620319399Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.693192_123.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.693192891Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.699710_124.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.699710089Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210283.829823_36.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.829823147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.834483_39.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.834483709Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.836936_40.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.836936436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.840722_41.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.840722909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.861648_42.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.861648232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.866712_43.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.866712671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.868791_44.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.868791626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.872744_45.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.872744406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.912794_46.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.912794447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.917145_47.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.917145151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.919062_48.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.919062388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.922951_49.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.922951366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.605197_57.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.605197827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.609960_63.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.609960143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.611973_64.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.611973570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.616421_65.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.616421099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.639658_66.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.639658316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.644456_67.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.644456209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.646533_68.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.646533915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.650631_69.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.650631154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.686421_70.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.686421776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.690333_71.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.690333849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.692080_72.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.692080678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.695551_73.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.695551020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.726081_74.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.726081643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.730388_90.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.730388928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.732439_91.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.732439720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.736490_92.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.736490843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.761836_93.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.761836097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.766320_94.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.766320855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.768185_95.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.768185140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.772392_96.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.772392113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.815999_97.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.815999846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.820609_98.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.820609109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.822670_99.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.822670869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.826529_100.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.826529286Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.848864_101.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.848864850Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.864470_102.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.864470281Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.868456_103.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.868456806Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.872836_104.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.872836675Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.886607_105.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.886607990Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.893519_106.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.893519500Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210554.595214_107.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.595214941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.599281_144.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.599281417Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.601295_145.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.601295577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.605442_146.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.605442561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.611010_147.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.611010048Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"write"}}},{"id":"1_1591210554.616959_148.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.616959335Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210672.236763_164.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.236763068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.241777_178.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.241777217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210672.243544_179.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.243544464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.247152_180.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.247152542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210678.159531_181.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.159531729Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"write"}}},{"id":"1_1591210678.170340_188.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.170340522Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210742.802927_200.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.802927214Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"write"}}},{"id":"1_1591210742.813132_206.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.813132555Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210781.307965_207.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.307965472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.312070_220.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.312070631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.313936_221.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.313936390Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.317822_222.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.317822165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.341024_223.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.341024973Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.345840_224.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.345840769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.348034_225.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.348034677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.352426_226.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.352426848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.392769_227.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.392769566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.397143_228.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.397143855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.399135_229.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.399135264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.403065_230.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.403065260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.425948_231.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.425948130Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.432382_232.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.432382276Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.437135_233.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.437135275Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.443776_234.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.443776157Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.455906_235.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.455906662Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.464609_236.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.464609666Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.113774_237.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.113774215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.118343_254.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.118343820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.120325_255.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.120325874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.124318_256.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.124318849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.150068_257.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.150068619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.154668_258.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.154668276Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.156826_259.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.156826084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.160460_260.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.160460106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.203944_261.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.203944450Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.208516_262.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.208516698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.210607_263.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.210607287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.214106_264.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.214106000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.234619_265.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.234619010Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.241328_266.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.241328517Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.244971_267.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.244971814Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.251310_268.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.251310468Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.255948_269.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.255948064Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.263326_270.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.263326737Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.278680_271.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.278680379Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.295668_272.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.295668458Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.315521_273.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.315521132Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.335215_274.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.335215505Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.344022_275.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.344022794Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.348893_276.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.348893312Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210812.495863_277.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.495863761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.501345_294.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.501345043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.503316_295.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.503316997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.506988_296.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.506988748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.535091_297.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.535091796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.539087_298.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.539087230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.541097_299.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.541097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.544897_300.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.544897592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.584466_301.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.584466446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.588759_302.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.588759294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.590886_303.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.590886925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.594792_304.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.594792502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.614851_305.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.614851514Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.619870_306.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.619870881Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.669035_307.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.669035441Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.680146_308.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.680146030Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.740730_309.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.740730890Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.753891_310.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.753891023Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.826462_311.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.826462023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.831421_312.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.831421268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.833509_313.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.833509372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.837854_314.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.837854453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.867529_315.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.867529669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.872505_316.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.872505385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.874743_317.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.874743428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.880764_318.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.880764234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.941825_319.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.941825604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.945830_320.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.945830822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.947645_321.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.947645315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.951102_322.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.951102783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.971267_323.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.971267302Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.977341_324.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.977341117Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.026688_325.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.026688540Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.037067_326.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.037067265Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.098397_327.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.098397382Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.107261_328.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.107261184Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.172942_329.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.172942744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.177907_330.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.177907121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.180207_331.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.180207094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.184417_332.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.184417965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.210911_333.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.210911135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.215792_334.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.215792268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.218015_335.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.218015532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.222023_336.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.222023040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.267046_337.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.267046477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.273494_338.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.273494143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.275786_339.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.275786029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.284251_340.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.284251835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.313152_341.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.313152215Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.317982_342.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.317982989Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210814.365223_343.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.365223445Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"write"}}},{"id":"1_1591210814.377235_344.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.377235109Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.437780_345.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.437780713Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"write"}}},{"id":"1_1591210815.449664_346.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.449664785Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.524838_347.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.524838343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.532190_348.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.532190754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.534642_349.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.534642732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.539137_350.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.539137445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.564144_351.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.564144878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.568861_352.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.568861847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.570743_353.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.570743234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.575083_354.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.575083843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.614646_355.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.614646325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.619421_356.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.619421281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.621627_357.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.621627252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.625665_358.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.625665171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.195107_361.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.195107231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.199954_362.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.199954754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.201884_363.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.201884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.205979_364.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.205979436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.230298_365.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.230298606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.237270_366.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.237270769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.239886_367.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.239886049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.245344_369.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.245344655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.287745_371.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.287745221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.294791_372.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.294791503Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.297704_373.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.297704179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.301177_374.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.301177911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:34.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.331 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:34.369 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:34.370 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:34.627 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210125.646195_38.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.646195632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"write"}}},{"id":"1_1591210125.653449_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.653449077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.551191_41.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.551191174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.561114_44.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.561114974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.562915_45.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.562915114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.566240_46.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.566240457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.586275_47.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.586275825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.590554_48.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.590554050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.592599_49.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.592599034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.597559_50.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.597559865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.805687_51.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.805687792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.809012_52.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.809012918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.810514_53.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.810514753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.813548_54.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.813548628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.228196_55.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.228196711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.232832_57.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.232832116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.234688_58.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.234688598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.238737_59.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.238737032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.257320_60.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.257320059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.262348_61.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.262348221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.264759_62.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.264759978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.269414_63.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.269414349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.308125_64.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.308125120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.312149_65.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.312149093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.313967_66.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.313967428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.317549_67.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.317549729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.000602_68.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.000602707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.004506_70.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.004506898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.006174_71.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.006174782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.009372_72.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.009372170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.031860_73.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.031860332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.035738_74.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.035738385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.037460_75.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.037460166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.040767_76.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.040767522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.066084_77.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.066084318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.070207_78.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.070207165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.072017_79.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.072017297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.075542_80.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.075542707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.103016_81.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.103016563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.107358_82.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.107358242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.109115_83.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.109115163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.112561_84.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.112561397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.165315_85.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.165315641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.169519_86.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.169519322Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.171422_87.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.171422660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.174741_88.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.174741498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.266352_89.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.266352000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.270156_90.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.270156689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.271931_91.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.271931107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.275233_92.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.275233693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.369098_93.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.369098464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.373277_94.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.373277856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.375072_95.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.375072710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.378378_96.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.378378667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.421365_97.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.421365255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.424919_98.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.424919143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.426516_99.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.426516831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.429676_100.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.429676437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.519909_101.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.519909980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.526123_102.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.526123681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.528147_103.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.528147872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.531714_104.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.531714552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210276.048639_105.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.048639896Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.053020_107.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.053020734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.054976_108.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.054976082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.058926_109.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.058926026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.082565_110.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.082565090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.087382_111.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.087382132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.089688_112.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.089688894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.093302_113.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.093302341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.133481_114.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.133481597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.138334_115.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.138334193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.140074_116.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.140074226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.144087_117.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.144087557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210282.550024_118.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.550024390Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.556399_120.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.556399419Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.613019_121.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.613019625Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.620319_122.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.620319399Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.693192_123.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.693192891Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.699710_124.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.699710089Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210283.829823_36.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.829823147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.834483_39.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.834483709Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.836936_40.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.836936436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.840722_41.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.840722909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.861648_42.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.861648232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.866712_43.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.866712671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.868791_44.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.868791626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.872744_45.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.872744406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.912794_46.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.912794447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.917145_47.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.917145151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.919062_48.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.919062388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.922951_49.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.922951366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.605197_57.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.605197827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.609960_63.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.609960143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.611973_64.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.611973570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.616421_65.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.616421099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.639658_66.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.639658316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.644456_67.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.644456209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.646533_68.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.646533915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.650631_69.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.650631154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.686421_70.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.686421776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.690333_71.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.690333849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.692080_72.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.692080678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.695551_73.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.695551020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.726081_74.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.726081643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.730388_90.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.730388928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.732439_91.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.732439720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.736490_92.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.736490843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.761836_93.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.761836097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.766320_94.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.766320855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.768185_95.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.768185140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.772392_96.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.772392113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.815999_97.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.815999846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.820609_98.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.820609109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.822670_99.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.822670869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.826529_100.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.826529286Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.848864_101.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.848864850Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.864470_102.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.864470281Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.868456_103.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.868456806Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.872836_104.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.872836675Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.886607_105.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.886607990Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.893519_106.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.893519500Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210554.595214_107.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.595214941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.599281_144.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.599281417Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.601295_145.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.601295577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.605442_146.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.605442561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.611010_147.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.611010048Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"write"}}},{"id":"1_1591210554.616959_148.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.616959335Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210672.236763_164.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.236763068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.241777_178.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.241777217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210672.243544_179.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.243544464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.247152_180.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.247152542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210678.159531_181.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.159531729Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"write"}}},{"id":"1_1591210678.170340_188.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.170340522Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210742.802927_200.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.802927214Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"write"}}},{"id":"1_1591210742.813132_206.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.813132555Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210781.307965_207.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.307965472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.312070_220.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.312070631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.313936_221.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.313936390Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.317822_222.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.317822165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.341024_223.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.341024973Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.345840_224.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.345840769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.348034_225.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.348034677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.352426_226.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.352426848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.392769_227.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.392769566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.397143_228.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.397143855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.399135_229.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.399135264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.403065_230.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.403065260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.425948_231.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.425948130Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.432382_232.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.432382276Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.437135_233.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.437135275Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.443776_234.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.443776157Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.455906_235.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.455906662Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.464609_236.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.464609666Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.113774_237.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.113774215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.118343_254.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.118343820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.120325_255.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.120325874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.124318_256.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.124318849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.150068_257.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.150068619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.154668_258.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.154668276Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.156826_259.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.156826084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.160460_260.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.160460106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.203944_261.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.203944450Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.208516_262.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.208516698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.210607_263.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.210607287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.214106_264.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.214106000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.234619_265.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.234619010Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.241328_266.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.241328517Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.244971_267.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.244971814Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.251310_268.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.251310468Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.255948_269.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.255948064Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.263326_270.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.263326737Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.278680_271.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.278680379Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.295668_272.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.295668458Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.315521_273.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.315521132Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.335215_274.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.335215505Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.344022_275.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.344022794Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.348893_276.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.348893312Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210812.495863_277.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.495863761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.501345_294.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.501345043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.503316_295.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.503316997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.506988_296.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.506988748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.535091_297.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.535091796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.539087_298.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.539087230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.541097_299.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.541097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.544897_300.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.544897592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.584466_301.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.584466446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.588759_302.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.588759294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.590886_303.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.590886925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.594792_304.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.594792502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.614851_305.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.614851514Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.619870_306.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.619870881Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.669035_307.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.669035441Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.680146_308.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.680146030Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.740730_309.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.740730890Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.753891_310.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.753891023Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.826462_311.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.826462023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.831421_312.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.831421268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.833509_313.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.833509372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.837854_314.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.837854453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.867529_315.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.867529669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.872505_316.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.872505385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.874743_317.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.874743428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.880764_318.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.880764234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.941825_319.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.941825604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.945830_320.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.945830822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.947645_321.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.947645315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.951102_322.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.951102783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.971267_323.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.971267302Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.977341_324.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.977341117Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.026688_325.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.026688540Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.037067_326.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.037067265Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.098397_327.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.098397382Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.107261_328.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.107261184Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.172942_329.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.172942744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.177907_330.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.177907121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.180207_331.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.180207094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.184417_332.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.184417965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.210911_333.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.210911135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.215792_334.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.215792268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.218015_335.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.218015532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.222023_336.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.222023040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.267046_337.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.267046477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.273494_338.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.273494143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.275786_339.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.275786029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.284251_340.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.284251835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.313152_341.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.313152215Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.317982_342.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.317982989Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210814.365223_343.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.365223445Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"write"}}},{"id":"1_1591210814.377235_344.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.377235109Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.437780_345.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.437780713Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"write"}}},{"id":"1_1591210815.449664_346.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.449664785Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.524838_347.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.524838343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.532190_348.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.532190754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.534642_349.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.534642732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.539137_350.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.539137445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.564144_351.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.564144878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.568861_352.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.568861847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.570743_353.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.570743234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.575083_354.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.575083843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.614646_355.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.614646325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.619421_356.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.619421281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.621627_357.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.621627252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.625665_358.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.625665171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.195107_361.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.195107231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.199954_362.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.199954754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.201884_363.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.201884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.205979_364.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.205979436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.230298_365.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.230298606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.237270_366.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.237270769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.239886_367.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.239886049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.245344_369.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.245344655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.287745_371.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.287745221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.294791_372.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.294791503Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.297704_373.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.297704179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.301177_374.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.301177911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:34.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.631 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:34.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:34.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:34.920 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210125.646195_38.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.646195632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"write"}}},{"id":"1_1591210125.653449_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.653449077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.551191_41.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.551191174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.561114_44.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.561114974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.562915_45.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.562915114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.566240_46.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.566240457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.586275_47.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.586275825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.590554_48.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.590554050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.592599_49.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.592599034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.597559_50.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.597559865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.805687_51.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.805687792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.809012_52.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.809012918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.810514_53.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.810514753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.813548_54.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.813548628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.228196_55.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.228196711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.232832_57.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.232832116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.234688_58.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.234688598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.238737_59.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.238737032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.257320_60.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.257320059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.262348_61.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.262348221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.264759_62.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.264759978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.269414_63.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.269414349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.308125_64.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.308125120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.312149_65.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.312149093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.313967_66.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.313967428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.317549_67.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.317549729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.000602_68.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.000602707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.004506_70.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.004506898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.006174_71.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.006174782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.009372_72.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.009372170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.031860_73.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.031860332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.035738_74.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.035738385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.037460_75.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.037460166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.040767_76.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.040767522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.066084_77.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.066084318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.070207_78.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.070207165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.072017_79.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.072017297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.075542_80.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.075542707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.103016_81.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.103016563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.107358_82.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.107358242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.109115_83.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.109115163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.112561_84.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.112561397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.165315_85.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.165315641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.169519_86.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.169519322Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.171422_87.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.171422660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.174741_88.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.174741498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.266352_89.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.266352000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.270156_90.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.270156689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.271931_91.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.271931107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.275233_92.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.275233693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.369098_93.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.369098464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.373277_94.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.373277856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.375072_95.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.375072710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.378378_96.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.378378667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.421365_97.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.421365255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.424919_98.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.424919143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.426516_99.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.426516831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.429676_100.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.429676437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.519909_101.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.519909980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.526123_102.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.526123681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.528147_103.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.528147872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.531714_104.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.531714552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210276.048639_105.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.048639896Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.053020_107.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.053020734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.054976_108.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.054976082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.058926_109.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.058926026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.082565_110.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.082565090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.087382_111.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.087382132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.089688_112.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.089688894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.093302_113.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.093302341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.133481_114.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.133481597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.138334_115.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.138334193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.140074_116.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.140074226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.144087_117.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.144087557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210282.550024_118.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.550024390Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.556399_120.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.556399419Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.613019_121.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.613019625Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.620319_122.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.620319399Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.693192_123.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.693192891Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.699710_124.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.699710089Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210283.829823_36.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.829823147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.834483_39.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.834483709Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.836936_40.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.836936436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.840722_41.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.840722909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.861648_42.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.861648232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.866712_43.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.866712671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.868791_44.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.868791626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.872744_45.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.872744406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.912794_46.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.912794447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.917145_47.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.917145151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.919062_48.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.919062388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.922951_49.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.922951366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.605197_57.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.605197827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.609960_63.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.609960143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.611973_64.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.611973570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.616421_65.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.616421099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.639658_66.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.639658316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.644456_67.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.644456209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.646533_68.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.646533915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.650631_69.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.650631154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.686421_70.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.686421776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.690333_71.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.690333849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.692080_72.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.692080678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.695551_73.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.695551020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.726081_74.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.726081643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.730388_90.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.730388928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.732439_91.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.732439720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.736490_92.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.736490843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.761836_93.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.761836097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.766320_94.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.766320855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.768185_95.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.768185140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.772392_96.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.772392113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.815999_97.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.815999846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.820609_98.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.820609109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.822670_99.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.822670869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.826529_100.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.826529286Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.848864_101.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.848864850Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.864470_102.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.864470281Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.868456_103.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.868456806Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.872836_104.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.872836675Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.886607_105.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.886607990Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.893519_106.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.893519500Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210554.595214_107.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.595214941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.599281_144.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.599281417Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.601295_145.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.601295577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.605442_146.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.605442561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.611010_147.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.611010048Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"write"}}},{"id":"1_1591210554.616959_148.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.616959335Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210672.236763_164.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.236763068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.241777_178.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.241777217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210672.243544_179.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.243544464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.247152_180.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.247152542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210678.159531_181.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.159531729Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"write"}}},{"id":"1_1591210678.170340_188.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.170340522Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210742.802927_200.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.802927214Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"write"}}},{"id":"1_1591210742.813132_206.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.813132555Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210781.307965_207.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.307965472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.312070_220.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.312070631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.313936_221.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.313936390Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.317822_222.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.317822165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.341024_223.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.341024973Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.345840_224.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.345840769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.348034_225.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.348034677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.352426_226.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.352426848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.392769_227.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.392769566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.397143_228.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.397143855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.399135_229.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.399135264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.403065_230.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.403065260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.425948_231.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.425948130Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.432382_232.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.432382276Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.437135_233.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.437135275Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.443776_234.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.443776157Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.455906_235.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.455906662Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.464609_236.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.464609666Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.113774_237.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.113774215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.118343_254.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.118343820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.120325_255.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.120325874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.124318_256.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.124318849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.150068_257.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.150068619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.154668_258.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.154668276Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.156826_259.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.156826084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.160460_260.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.160460106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.203944_261.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.203944450Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.208516_262.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.208516698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.210607_263.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.210607287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.214106_264.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.214106000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.234619_265.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.234619010Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.241328_266.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.241328517Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.244971_267.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.244971814Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.251310_268.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.251310468Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.255948_269.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.255948064Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.263326_270.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.263326737Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.278680_271.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.278680379Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.295668_272.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.295668458Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.315521_273.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.315521132Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.335215_274.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.335215505Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.344022_275.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.344022794Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.348893_276.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.348893312Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210812.495863_277.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.495863761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.501345_294.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.501345043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.503316_295.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.503316997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.506988_296.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.506988748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.535091_297.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.535091796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.539087_298.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.539087230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.541097_299.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.541097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.544897_300.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.544897592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.584466_301.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.584466446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.588759_302.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.588759294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.590886_303.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.590886925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.594792_304.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.594792502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.614851_305.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.614851514Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.619870_306.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.619870881Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.669035_307.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.669035441Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.680146_308.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.680146030Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.740730_309.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.740730890Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.753891_310.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.753891023Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.826462_311.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.826462023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.831421_312.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.831421268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.833509_313.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.833509372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.837854_314.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.837854453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.867529_315.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.867529669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.872505_316.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.872505385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.874743_317.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.874743428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.880764_318.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.880764234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.941825_319.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.941825604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.945830_320.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.945830822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.947645_321.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.947645315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.951102_322.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.951102783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.971267_323.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.971267302Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.977341_324.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.977341117Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.026688_325.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.026688540Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.037067_326.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.037067265Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.098397_327.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.098397382Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.107261_328.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.107261184Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.172942_329.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.172942744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.177907_330.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.177907121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.180207_331.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.180207094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.184417_332.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.184417965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.210911_333.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.210911135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.215792_334.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.215792268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.218015_335.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.218015532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.222023_336.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.222023040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.267046_337.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.267046477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.273494_338.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.273494143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.275786_339.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.275786029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.284251_340.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.284251835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.313152_341.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.313152215Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.317982_342.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.317982989Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210814.365223_343.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.365223445Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"write"}}},{"id":"1_1591210814.377235_344.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.377235109Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.437780_345.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.437780713Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"write"}}},{"id":"1_1591210815.449664_346.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.449664785Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.524838_347.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.524838343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.532190_348.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.532190754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.534642_349.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.534642732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.539137_350.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.539137445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.564144_351.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.564144878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.568861_352.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.568861847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.570743_353.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.570743234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.575083_354.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.575083843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.614646_355.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.614646325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.619421_356.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.619421281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.621627_357.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.621627252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.625665_358.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.625665171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.195107_361.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.195107231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.199954_362.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.199954754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.201884_363.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.201884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.205979_364.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.205979436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.230298_365.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.230298606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.237270_366.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.237270769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.239886_367.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.239886049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.245344_369.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.245344655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.287745_371.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.287745221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.294791_372.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.294791503Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.297704_373.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.297704179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.301177_374.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.301177911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:34.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:34.924 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:34.967 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:34.968 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:35.232 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210125.646195_38.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.646195632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"write"}}},{"id":"1_1591210125.653449_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-03T18:48:45.653449077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xCDIlMY5XvxwEz4BlVAf-a0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.551191_41.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.551191174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.561114_44.1","section":"bucket.instance","name":"rhmmtn-1:12983a69-4226-40f7-aaec-4a03acd75b02.4472.1","timestamp":"2020-06-03T18:50:47.561114974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6DAkeCO_4YgE6ay84N9Z6jK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.562915_45.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.562915114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.566240_46.1","section":"bucket","name":"rhmmtn-1","timestamp":"2020-06-03T18:50:47.566240457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JQMVZbG0z6uWZqUqIixt3NG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.586275_47.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.586275825Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.590554_48.1","section":"bucket.instance","name":"rhmmtn-2:12983a69-4226-40f7-aaec-4a03acd75b02.4472.2","timestamp":"2020-06-03T18:50:47.590554050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6f6n_NOYhTCX4ZX7GVUn4W2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210247.592599_49.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.592599034Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"write"}}},{"id":"1_1591210247.597559_50.1","section":"bucket","name":"rhmmtn-2","timestamp":"2020-06-03T18:50:47.597559865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tSXAMCIYlqAriBWHvcC5v2O","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.805687_51.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.805687792Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.809012_52.1","section":"bucket.instance","name":"rhmmtn-3:12983a69-4226-40f7-aaec-4a03acd75b02.4472.3","timestamp":"2020-06-03T18:50:50.809012918Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rFFK2I1VlH20Xh9U3bFIriT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210250.810514_53.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.810514753Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"write"}}},{"id":"1_1591210250.813548_54.1","section":"bucket","name":"rhmmtn-3","timestamp":"2020-06-03T18:50:50.813548628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f_8Ovu_h1vqu3-tZ7whwt6C","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.228196_55.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.228196711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.232832_57.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.4","timestamp":"2020-06-03T18:50:57.232832116Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_g9pKCmxVyUjrJ7DnQZ_LtqA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.234688_58.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.234688598Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.238737_59.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:50:57.238737032Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XT0OlU7blNc9OChX5sxR9VW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.257320_60.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.257320059Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.262348_61.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.5","timestamp":"2020-06-03T18:50:57.262348221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xWKAK8vlg12D8Xox1KCepZu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.264759_62.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.264759978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.269414_63.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:50:57.269414349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_30OjAofZL339e5kl6PrLFkn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.308125_64.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.308125120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.312149_65.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.6","timestamp":"2020-06-03T18:50:57.312149093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_li5zg6FTrki0GcH-G4R4lZY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210257.313967_66.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.313967428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"write"}}},{"id":"1_1591210257.317549_67.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:50:57.317549729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m1IwV1ghzzqYqbEWiM6GQ5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.000602_68.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.000602707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.004506_70.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.7","timestamp":"2020-06-03T18:51:09.004506898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wpsKOIuFx47OTnB1IzoCRaf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.006174_71.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.006174782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.009372_72.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.009372170Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9VEKiE2wlf5HslLmGrD39c","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.031860_73.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.031860332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.035738_74.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.8","timestamp":"2020-06-03T18:51:09.035738385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_noYVz2MA0cbtVCzR3kFcAWo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.037460_75.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.037460166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.040767_76.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.040767522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iA8QV7ntAOPru1GgiF5pV8h","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.066084_77.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.066084318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.070207_78.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.9","timestamp":"2020-06-03T18:51:09.070207165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OaiA7Lk70r3NGpX1CyIlM21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.072017_79.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.072017297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.075542_80.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.075542707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SnJ-8LLLYYUdScTz7M-0T0T","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.103016_81.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.103016563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.107358_82.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.10","timestamp":"2020-06-03T18:51:09.107358242Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GlolIDZCEW03oBHLVxQfQfm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.109115_83.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.109115163Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.112561_84.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.112561397Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wjn183TPUe7brw_dFJ651W_","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.165315_85.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.165315641Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.169519_86.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.11","timestamp":"2020-06-03T18:51:09.169519322Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dmMWLH8WWIvAXTTzDVi2UOO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.171422_87.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.171422660Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.174741_88.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.174741498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_02Fi-eqNHZ6FLUkSgdjcm86","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.266352_89.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.266352000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.270156_90.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.12","timestamp":"2020-06-03T18:51:09.270156689Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nCydMoradtT4MqVUA2yEVxF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.271931_91.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.271931107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.275233_92.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.275233693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YY8Qt0nXtYE1bj6sZmCxep3","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.369098_93.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.369098464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.373277_94.1","section":"bucket.instance","name":"rhmmtn-4:12983a69-4226-40f7-aaec-4a03acd75b02.4472.13","timestamp":"2020-06-03T18:51:09.373277856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7YQsb_2S2q_TGoMqHCtNoQ3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.375072_95.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.375072710Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.378378_96.1","section":"bucket","name":"rhmmtn-4","timestamp":"2020-06-03T18:51:09.378378667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p3RYWsK2Lh_qlmL5D20SSGo","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.421365_97.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.421365255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.424919_98.1","section":"bucket.instance","name":"rhmmtn-5:12983a69-4226-40f7-aaec-4a03acd75b02.4472.14","timestamp":"2020-06-03T18:51:09.424919143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_POs3xoK6PYjcTWNjlJXe2po","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.426516_99.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.426516831Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.429676_100.1","section":"bucket","name":"rhmmtn-5","timestamp":"2020-06-03T18:51:09.429676437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lTGxxmu5gFz5V2lVn55U4Iw","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210269.519909_101.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.519909980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.526123_102.1","section":"bucket.instance","name":"rhmmtn-6:12983a69-4226-40f7-aaec-4a03acd75b02.4472.15","timestamp":"2020-06-03T18:51:09.526123681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0jv932KzzUPOgjM6JqfpPPR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210269.528147_103.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.528147872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210269.531714_104.1","section":"bucket","name":"rhmmtn-6","timestamp":"2020-06-03T18:51:09.531714552Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G3Rxm4N5Y48uwFnNdVkEXXA","ver":1},"status":{"status":"abort"}}},{"id":"1_1591210276.048639_105.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.048639896Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.053020_107.1","section":"bucket.instance","name":"rhmmtn-7:12983a69-4226-40f7-aaec-4a03acd75b02.4472.16","timestamp":"2020-06-03T18:51:16.053020734Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onL7oW4XFBLzgHHWa47kypR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.054976_108.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.054976082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.058926_109.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:16.058926026Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.082565_110.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.082565090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.087382_111.1","section":"bucket.instance","name":"rhmmtn-8:12983a69-4226-40f7-aaec-4a03acd75b02.4472.17","timestamp":"2020-06-03T18:51:16.087382132Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iUNsegvvmkjOxOpZ4mlrWG8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.089688_112.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.089688894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.093302_113.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:16.093302341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.133481_114.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.133481597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.138334_115.1","section":"bucket.instance","name":"rhmmtn-9:12983a69-4226-40f7-aaec-4a03acd75b02.4472.18","timestamp":"2020-06-03T18:51:16.138334193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LrTwpS2D0sKa4mR_r86WOrm","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210276.140074_116.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.140074226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210276.144087_117.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:16.144087557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210282.550024_118.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.550024390Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.556399_120.1","section":"bucket","name":"rhmmtn-7","timestamp":"2020-06-03T18:51:22.556399419Z","data":{"read_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":1},"write_version":{"tag":"_5W5bpAAOcV2SSEM5Eyy7kEf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.613019_121.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.613019625Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.620319_122.1","section":"bucket","name":"rhmmtn-8","timestamp":"2020-06-03T18:51:22.620319399Z","data":{"read_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":1},"write_version":{"tag":"_2Qdk9v8N6PXSQRswRaWmygH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210282.693192_123.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.693192891Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"remove"}}},{"id":"1_1591210282.699710_124.1","section":"bucket","name":"rhmmtn-9","timestamp":"2020-06-03T18:51:22.699710089Z","data":{"read_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":1},"write_version":{"tag":"_CLKCtcGwnjh8lm5fVEHV1YG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210283.829823_36.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.829823147Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.834483_39.1","section":"bucket.instance","name":"rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19","timestamp":"2020-06-03T18:51:23.834483709Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qm8WIMJWvNexfDl7842_GRr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.836936_40.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.836936436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.840722_41.1","section":"bucket","name":"rhmmtn-10","timestamp":"2020-06-03T18:51:23.840722909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_78p6fpFeayvasvYD6h3wDxJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.861648_42.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.861648232Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.866712_43.1","section":"bucket.instance","name":"rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20","timestamp":"2020-06-03T18:51:23.866712671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__PFz9xTbUsRDKtiUxsFumaH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.868791_44.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.868791626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.872744_45.1","section":"bucket","name":"rhmmtn-11","timestamp":"2020-06-03T18:51:23.872744406Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ZOQA5QING1n23ZKkUqfTNd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.912794_46.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.912794447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.917145_47.1","section":"bucket.instance","name":"rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21","timestamp":"2020-06-03T18:51:23.917145151Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RPoSmY8NL_oAvXTabUj2xOT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210283.919062_48.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.919062388Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"write"}}},{"id":"1_1591210283.922951_49.1","section":"bucket","name":"rhmmtn-12","timestamp":"2020-06-03T18:51:23.922951366Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K7a-qFyaabegI6fK-pNsQo5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.605197_57.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.605197827Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.609960_63.1","section":"bucket.instance","name":"rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22","timestamp":"2020-06-03T18:51:36.609960143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EDXU0-AmQOToOn-s6Wqw6uQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.611973_64.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.611973570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.616421_65.1","section":"bucket","name":"rhmmtn-13","timestamp":"2020-06-03T18:51:36.616421099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_om3oQLw5DesUCvJzWQc4rhZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.639658_66.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.639658316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.644456_67.1","section":"bucket.instance","name":"rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23","timestamp":"2020-06-03T18:51:36.644456209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORPsmSd8f5yWUbFflQOoSa-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.646533_68.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.646533915Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.650631_69.1","section":"bucket","name":"rhmmtn-14","timestamp":"2020-06-03T18:51:36.650631154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V3aCSC8fh9M2wjP5QtCArHn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.686421_70.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.686421776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.690333_71.1","section":"bucket.instance","name":"rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24","timestamp":"2020-06-03T18:51:36.690333849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ku8mMXilthnDh-E6yFGQlPT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210296.692080_72.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.692080678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"write"}}},{"id":"1_1591210296.695551_73.1","section":"bucket","name":"rhmmtn-15","timestamp":"2020-06-03T18:51:36.695551020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iPK0BWAj9beH6VkXulIP0Tx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.726081_74.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.726081643Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.730388_90.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.730388928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.732439_91.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.732439720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.736490_92.1","section":"bucket","name":"rhmmtn-16","timestamp":"2020-06-03T18:52:25.736490843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ikC6Z6HCYExS157K0uz5Uv1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.761836_93.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.761836097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.766320_94.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.766320855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.768185_95.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.768185140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.772392_96.1","section":"bucket","name":"rhmmtn-17","timestamp":"2020-06-03T18:52:25.772392113Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eAatGMZ9sZ36lv39B4jUj-F","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.815999_97.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.815999846Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.820609_98.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.820609109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.822670_99.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.822670869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210345.826529_100.1","section":"bucket","name":"rhmmtn-18","timestamp":"2020-06-03T18:52:25.826529286Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XKjwbkD0-QLkONA0R_Y-HLZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210345.848864_101.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.848864850Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.864470_102.1","section":"bucket.instance","name":"rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25","timestamp":"2020-06-03T18:52:25.864470281Z","data":{"read_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":1},"write_version":{"tag":"_62on8Jl_iUnwDNf33bUeSaX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.868456_103.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.868456806Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.872836_104.1","section":"bucket.instance","name":"rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26","timestamp":"2020-06-03T18:52:25.872836675Z","data":{"read_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":1},"write_version":{"tag":"_Wzf7_Gnh3iF7ZQ7969ZHXzV","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210345.886607_105.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.886607990Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210345.893519_106.1","section":"bucket.instance","name":"rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27","timestamp":"2020-06-03T18:52:25.893519500Z","data":{"read_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":1},"write_version":{"tag":"_q9jVrOf7Re-OAkJkkifYdKg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210554.595214_107.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.595214941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.599281_144.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.599281417Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.601295_145.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.601295577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210554.605442_146.1","section":"bucket","name":"rhmmtn-19","timestamp":"2020-06-03T18:55:54.605442561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5FbGWV7a2LudXU_NFVDaitv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210554.611010_147.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.611010048Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"write"}}},{"id":"1_1591210554.616959_148.1","section":"bucket.instance","name":"rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28","timestamp":"2020-06-03T18:55:54.616959335Z","data":{"read_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":1},"write_version":{"tag":"_dC-gdM_nXtMBClBmlkwkEkA","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210672.236763_164.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.236763068Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.241777_178.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:52.241777217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210672.243544_179.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.243544464Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"write"}}},{"id":"1_1591210672.247152_180.1","section":"bucket","name":"rhmmtn-20","timestamp":"2020-06-03T18:57:52.247152542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4_1S2aNwnUaZKTGNCMilso","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210678.159531_181.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.159531729Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"write"}}},{"id":"1_1591210678.170340_188.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:57:58.170340522Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":1},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210742.802927_200.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.802927214Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"write"}}},{"id":"1_1591210742.813132_206.1","section":"bucket.instance","name":"rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29","timestamp":"2020-06-03T18:59:02.813132555Z","data":{"read_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":2},"write_version":{"tag":"_Evcczw6v4uH1h7tOp2JHISk","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210781.307965_207.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.307965472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.312070_220.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.312070631Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.313936_221.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.313936390Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.317822_222.1","section":"bucket","name":"rhmmtn-21","timestamp":"2020-06-03T18:59:41.317822165Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gBpT6uYm2j0IuumbcwuQgqT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.341024_223.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.341024973Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.345840_224.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.345840769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.348034_225.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.348034677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.352426_226.1","section":"bucket","name":"rhmmtn-22","timestamp":"2020-06-03T18:59:41.352426848Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ruy0XnfU4qyt7iW_tLtjlF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.392769_227.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.392769566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.397143_228.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.397143855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.399135_229.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.399135264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210781.403065_230.1","section":"bucket","name":"rhmmtn-23","timestamp":"2020-06-03T18:59:41.403065260Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qzHPhrtYk3R0tGKbisXCRHD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210781.425948_231.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.425948130Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.432382_232.1","section":"bucket.instance","name":"rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30","timestamp":"2020-06-03T18:59:41.432382276Z","data":{"read_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":1},"write_version":{"tag":"_pKOvOt3hH4sRQgdUSx1rQbL","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.437135_233.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.437135275Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.443776_234.1","section":"bucket.instance","name":"rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31","timestamp":"2020-06-03T18:59:41.443776157Z","data":{"read_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":1},"write_version":{"tag":"_1VmjCHgjP2eCSXIL_SPc-Wg","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210781.455906_235.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.455906662Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"write"}}},{"id":"1_1591210781.464609_236.1","section":"bucket.instance","name":"rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32","timestamp":"2020-06-03T18:59:41.464609666Z","data":{"read_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":1},"write_version":{"tag":"_vkHJOKFslxMmwNNW9PGv7Dw","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.113774_237.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.113774215Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.118343_254.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.118343820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.120325_255.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.120325874Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.124318_256.1","section":"bucket","name":"rhmmtn-24","timestamp":"2020-06-03T18:59:57.124318849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GdXlZNn0DklhoaipYzwkWwb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.150068_257.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.150068619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.154668_258.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.154668276Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.156826_259.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.156826084Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.160460_260.1","section":"bucket","name":"rhmmtn-25","timestamp":"2020-06-03T18:59:57.160460106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9_4LB3Mqk049vw3PpoLoi3Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.203944_261.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.203944450Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.208516_262.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.208516698Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.210607_263.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.210607287Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"write"}}},{"id":"1_1591210797.214106_264.1","section":"bucket","name":"rhmmtn-26","timestamp":"2020-06-03T18:59:57.214106000Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BqrKmBq3393qH5KV4egnr56","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210797.234619_265.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.234619010Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.241328_266.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.241328517Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":1},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.244971_267.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.244971814Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.251310_268.1","section":"bucket.instance","name":"rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33","timestamp":"2020-06-03T18:59:57.251310468Z","data":{"read_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":2},"write_version":{"tag":"_0Gu1HvK8Vs7T6CAsor8eySj","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.255948_269.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.255948064Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.263326_270.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.263326737Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":1},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.278680_271.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.278680379Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.295668_272.1","section":"bucket.instance","name":"rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34","timestamp":"2020-06-03T18:59:57.295668458Z","data":{"read_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":2},"write_version":{"tag":"_l8URnGeP68iTbrzzVMyixiP","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210797.315521_273.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.315521132Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"write"}}},{"id":"1_1591210797.335215_274.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.335215505Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":1},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210797.344022_275.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.344022794Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"write"}}},{"id":"1_1591210797.348893_276.1","section":"bucket.instance","name":"rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35","timestamp":"2020-06-03T18:59:57.348893312Z","data":{"read_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":2},"write_version":{"tag":"_PfJ6J0JLNjYolD7EEtoOieH","ver":3},"status":{"status":"complete"}}},{"id":"1_1591210812.495863_277.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.495863761Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.501345_294.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.501345043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.503316_295.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.503316997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.506988_296.1","section":"bucket","name":"rhmmtn-27","timestamp":"2020-06-03T19:00:12.506988748Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PL8EBgkhL6qXsHwzHJY8Ygr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.535091_297.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.535091796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.539087_298.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.539087230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.541097_299.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.541097002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.544897_300.1","section":"bucket","name":"rhmmtn-28","timestamp":"2020-06-03T19:00:12.544897592Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_onSHepgZzTyFxBTZWgEpdHi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.584466_301.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.584466446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.588759_302.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.588759294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.590886_303.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.590886925Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.594792_304.1","section":"bucket","name":"rhmmtn-29","timestamp":"2020-06-03T19:00:12.594792502Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ocGHI2XrkxfdBRt5OR40zH_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.614851_305.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.614851514Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.619870_306.1","section":"bucket.instance","name":"rhmmtn-27:12983a69-4226-40f7-aaec-4a03acd75b02.4472.36","timestamp":"2020-06-03T19:00:12.619870881Z","data":{"read_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":1},"write_version":{"tag":"_VrU0IbeBa-umd72D7M3IT00","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.669035_307.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.669035441Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.680146_308.1","section":"bucket.instance","name":"rhmmtn-28:12983a69-4226-40f7-aaec-4a03acd75b02.4472.37","timestamp":"2020-06-03T19:00:12.680146030Z","data":{"read_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":1},"write_version":{"tag":"_RAejFhGyrZpfuSE2xXGl0WX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.740730_309.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.740730890Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.753891_310.1","section":"bucket.instance","name":"rhmmtn-29:12983a69-4226-40f7-aaec-4a03acd75b02.4472.38","timestamp":"2020-06-03T19:00:12.753891023Z","data":{"read_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":1},"write_version":{"tag":"_yPuiMxKgh4XBwrBKxgPyNaO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210812.826462_311.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.826462023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.831421_312.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.831421268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.833509_313.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.833509372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.837854_314.1","section":"bucket","name":"rhmmtn-30","timestamp":"2020-06-03T19:00:12.837854453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7fUKiZQoB5wa8xtJLgYP9vZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.867529_315.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.867529669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.872505_316.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:12.872505385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.874743_317.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.874743428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.880764_318.1","section":"bucket","name":"rhmmtn-31","timestamp":"2020-06-03T19:00:12.880764234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-tL2oFJVLWnXD8hGgYCFA_f","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.941825_319.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.941825604Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.945830_320.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:12.945830822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.947645_321.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.947645315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"write"}}},{"id":"1_1591210812.951102_322.1","section":"bucket","name":"rhmmtn-32","timestamp":"2020-06-03T19:00:12.951102783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__LubkqN1l1069YxQjemGYm_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210812.971267_323.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.971267302Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"write"}}},{"id":"1_1591210812.977341_324.1","section":"bucket.instance","name":"rhmmtn-30:12983a69-4226-40f7-aaec-4a03acd75b02.4472.39","timestamp":"2020-06-03T19:00:12.977341117Z","data":{"read_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":1},"write_version":{"tag":"__UQdRr4xZnw3iQIhg5N_cgh","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.026688_325.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.026688540Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.037067_326.1","section":"bucket.instance","name":"rhmmtn-31:12983a69-4226-40f7-aaec-4a03acd75b02.4472.40","timestamp":"2020-06-03T19:00:13.037067265Z","data":{"read_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":1},"write_version":{"tag":"_Cjw7MsrrcguQMVx8YPeyrdn","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.098397_327.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.098397382Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.107261_328.1","section":"bucket.instance","name":"rhmmtn-32:12983a69-4226-40f7-aaec-4a03acd75b02.4472.41","timestamp":"2020-06-03T19:00:13.107261184Z","data":{"read_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":1},"write_version":{"tag":"_OvctSltc7WYaxssg0yBSlsO","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210813.172942_329.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.172942744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.177907_330.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.177907121Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.180207_331.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.180207094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.184417_332.1","section":"bucket","name":"rhmmtn-33","timestamp":"2020-06-03T19:00:13.184417965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fQ7UsfQfm-zlB8K-FUt_pEt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.210911_333.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.210911135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.215792_334.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:13.215792268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.218015_335.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.218015532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.222023_336.1","section":"bucket","name":"rhmmtn-34","timestamp":"2020-06-03T19:00:13.222023040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CuBBiH24ja53M3-W_ZhR_iy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.267046_337.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.267046477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.273494_338.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:13.273494143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.275786_339.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.275786029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"write"}}},{"id":"1_1591210813.284251_340.1","section":"bucket","name":"rhmmtn-35","timestamp":"2020-06-03T19:00:13.284251835Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2V4Omd7ZyMxysIEjMYGAF1P","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210813.313152_341.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.313152215Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"write"}}},{"id":"1_1591210813.317982_342.1","section":"bucket.instance","name":"rhmmtn-33:12983a69-4226-40f7-aaec-4a03acd75b02.4472.42","timestamp":"2020-06-03T19:00:13.317982989Z","data":{"read_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":1},"write_version":{"tag":"_fWZYWvBh67qhLwKza-hJNI3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210814.365223_343.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.365223445Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"write"}}},{"id":"1_1591210814.377235_344.1","section":"bucket.instance","name":"rhmmtn-34:12983a69-4226-40f7-aaec-4a03acd75b02.4472.43","timestamp":"2020-06-03T19:00:14.377235109Z","data":{"read_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":1},"write_version":{"tag":"_EogarBvuDUbn7ieQDgofPyG","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.437780_345.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.437780713Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"write"}}},{"id":"1_1591210815.449664_346.1","section":"bucket.instance","name":"rhmmtn-35:12983a69-4226-40f7-aaec-4a03acd75b02.4472.44","timestamp":"2020-06-03T19:00:15.449664785Z","data":{"read_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":1},"write_version":{"tag":"_srQkW9RU4ZzwQCJ51sFBkqD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591210815.524838_347.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.524838343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.532190_348.1","section":"bucket.instance","name":"rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45","timestamp":"2020-06-03T19:00:15.532190754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gXr1VlHmaU93NvUg2ekqK2U","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.534642_349.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.534642732Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.539137_350.1","section":"bucket","name":"rhmmtn-36","timestamp":"2020-06-03T19:00:15.539137445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qUPv9q0_g33OSQP5yqV6dlW","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.564144_351.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.564144878Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.568861_352.1","section":"bucket.instance","name":"rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46","timestamp":"2020-06-03T19:00:15.568861847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Qd4lEetDp_xEb1nf-DfFQmi","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.570743_353.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.570743234Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.575083_354.1","section":"bucket","name":"rhmmtn-37","timestamp":"2020-06-03T19:00:15.575083843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_61Fix89Eku-Aiiqc7DokJ9Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.614646_355.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.614646325Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.619421_356.1","section":"bucket.instance","name":"rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47","timestamp":"2020-06-03T19:00:15.619421281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FEYBbB2G7XedF9GVJNeQbv-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210815.621627_357.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.621627252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"write"}}},{"id":"1_1591210815.625665_358.1","section":"bucket","name":"rhmmtn-38","timestamp":"2020-06-03T19:00:15.625665171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Wr571WmvcmcVvdBR_RZPfJI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.195107_361.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.195107231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.199954_362.1","section":"bucket.instance","name":"rhmmtn-39:12983a69-4226-40f7-aaec-4a03acd75b02.4472.48","timestamp":"2020-06-03T19:00:17.199954754Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fRFSqies5YhlXysEWsGsKfC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.201884_363.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.201884493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.205979_364.1","section":"bucket","name":"rhmmtn-39","timestamp":"2020-06-03T19:00:17.205979436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C83kl-CANm-6dM0W5uay4QS","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.230298_365.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.230298606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.237270_366.1","section":"bucket.instance","name":"rhmmtn-40:12983a69-4226-40f7-aaec-4a03acd75b02.4472.49","timestamp":"2020-06-03T19:00:17.237270769Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Cmei-aeMEJmQLruO5tVoo1y","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.239886_367.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.239886049Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.245344_369.1","section":"bucket","name":"rhmmtn-40","timestamp":"2020-06-03T19:00:17.245344655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_D5WZVTlTZxy_vozwZZwrdAv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.287745_371.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.287745221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.294791_372.1","section":"bucket.instance","name":"rhmmtn-41:12983a69-4226-40f7-aaec-4a03acd75b02.4472.50","timestamp":"2020-06-03T19:00:17.294791503Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qgDQBc8Q_g21lkbY4cYdRvM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210817.297704_373.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.297704179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"write"}}},{"id":"1_1591210817.301177_374.1","section":"bucket","name":"rhmmtn-41","timestamp":"2020-06-03T19:00:17.301177911Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pAl7sFcVY584hZ6nSVEU_et","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:35.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.237 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:35.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:35.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:35.514 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210885.004279_211.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.004279606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.008806_212.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.008806494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.010717_213.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.010717416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.014378_214.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.014378726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.055808_215.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.055808683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.060330_216.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.060330568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.062171_217.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.062171085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.066318_218.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.066318940Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:35.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.516 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:35.789 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210885.004279_211.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.004279606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.008806_212.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.008806494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.010717_213.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.010717416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.014378_214.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.014378726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.055808_215.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.055808683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.060330_216.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.060330568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.062171_217.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.062171085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.066318_218.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.066318940Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:35.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:35.792 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:35.831 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:35.831 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:36.063 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210885.004279_211.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.004279606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.008806_212.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.008806494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.010717_213.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.010717416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.014378_214.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.014378726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.055808_215.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.055808683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.060330_216.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.060330568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.062171_217.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.062171085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.066318_218.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.066318940Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:36.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.064 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:36.102 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:36.102 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:36.352 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210885.004279_211.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.004279606Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.008806_212.1","section":"bucket.instance","name":"rhmmtn-42:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1","timestamp":"2020-06-03T19:01:25.008806494Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3xdHZG1gKyeBX36uEujh-Ar","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.010717_213.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.010717416Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.014378_214.1","section":"bucket","name":"rhmmtn-42","timestamp":"2020-06-03T19:01:25.014378726Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vSNm9En42YsrpnNhesvMXty","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.055808_215.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.055808683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.060330_216.1","section":"bucket.instance","name":"rhmmtn-43:3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2","timestamp":"2020-06-03T19:01:25.060330568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_66n_eOoReo_CzmbMa7nGBWj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210885.062171_217.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.062171085Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"write"}}},{"id":"1_1591210885.066318_218.1","section":"bucket","name":"rhmmtn-43","timestamp":"2020-06-03T19:01:25.066318940Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VoREfo-oEpUsfE_8BrdmBAJ","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:36.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.354 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:36.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:36.618 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210946.153944_133.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.153944685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.158811_134.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.158811495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.160783_135.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.160783540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.164651_136.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.164651185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.208832_137.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.208832244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.214871_138.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.214871424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.216619_139.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.216619143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.220397_140.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.220397279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:36.618 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.619 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:36.878 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210946.153944_133.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.153944685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.158811_134.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.158811495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.160783_135.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.160783540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.164651_136.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.164651185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.208832_137.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.208832244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.214871_138.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.214871424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.216619_139.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.216619143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.220397_140.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.220397279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:36.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:36.880 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:36.917 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:36.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:37.159 INFO:teuthology.orchestra.run.smithi112.stdout:[{"id":"1_1591210946.153944_133.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.153944685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.158811_134.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.158811495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.160783_135.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.160783540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.164651_136.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.164651185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.208832_137.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.208832244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.214871_138.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.214871424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.216619_139.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.216619143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.220397_140.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.220397279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:37.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:37.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:37.160 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:37.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:37.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:37.438 INFO:teuthology.orchestra.run.smithi061.stdout:[{"id":"1_1591210946.153944_133.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.153944685Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.158811_134.1","section":"bucket.instance","name":"rhmmtn-44:12983a69-4226-40f7-aaec-4a03acd75b02.7269.1","timestamp":"2020-06-03T19:02:26.158811495Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5qEw6B4sUEHzn6plcdEaF4R","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.160783_135.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.160783540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.164651_136.1","section":"bucket","name":"rhmmtn-44","timestamp":"2020-06-03T19:02:26.164651185Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_46_2HsXpOpqTdADLE5IIb21","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.208832_137.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.208832244Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.214871_138.1","section":"bucket.instance","name":"rhmmtn-45:12983a69-4226-40f7-aaec-4a03acd75b02.7269.2","timestamp":"2020-06-03T19:02:26.214871424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_A49_PvVl1OgfExC3vmTad_q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591210946.216619_139.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.216619143Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"write"}}},{"id":"1_1591210946.220397_140.1","section":"bucket","name":"rhmmtn-45","timestamp":"2020-06-03T19:02:26.220397279Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4w28CGPkU0bNCRCv8SJSXnV","ver":1},"status":{"status":"complete"}}}] 2020-06-03T19:02:37.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:37.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:37.439 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:37.475 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:37.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:37.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:37.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:37.748 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:37.789 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:37.790 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:38.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:38.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:38.165 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:38.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:38.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:38.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:38.450 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:38.488 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:38.488 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:38.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:38.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:38.861 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:38.900 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:38.901 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:39.287 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:39.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.290 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:39.329 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:39.330 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:39.583 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:39.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.585 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:39.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:39.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:39.856 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:39.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f641ddd2-34e8-45d4-a5d7-f7dfb77827d9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:39.858 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f641ddd2-34e8-45d4-a5d7-f7dfb77827d9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:39.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:39.899 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:40.155 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:40.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.156 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:40.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:40.194 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:40.428 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:40.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.430 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:40.469 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:40.470 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:40.722 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:40.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.724 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:40.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:40.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:40.995 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:40.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c667fc06-9ec1-4c12-8825-c7c36661d280', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:40.996 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c667fc06-9ec1-4c12-8825-c7c36661d280 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:41.036 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:41.036 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:41.285 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:41.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.287 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:41.326 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:41.326 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:41.553 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:41.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.554 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:41.825 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:41.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:41.827 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:41.865 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:41.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:42.114 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:42.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'b2e295e4-c2ab-4a12-b669-6be057e86c4f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.115 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period b2e295e4-c2ab-4a12-b669-6be057e86c4f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:42.154 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:42.154 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:42.404 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:42.405 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-06-03T19:02:42.406 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-46 2020-06-03T19:02:42.446 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-47 2020-06-03T19:02:42.511 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-48 2020-06-03T19:02:42.639 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:02:42.637+0000 7fcc4b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:02:42.640 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:02:42.634+0000 7fb2a77fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:02:42.642 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:02:42.641+0000 7fc3a2ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:02:42.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.715 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:42.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:42.750 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:42.945 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210962.828994_265.1","last_update":"2020-06-03T19:02:42.828994Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"1_1591210817.253304_370.1","last_update":"2020-06-03T19:00:17.253304Z"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T19:02:42.946 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:02:42.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:42.947 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:42.999 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:42.999 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:43.230 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.828994_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.828994Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:43.231 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:02:43.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.232 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:43.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:43.505 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210962.828994_265.1","last_update":"2020-06-03T19:02:42.828994Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"1_1591210817.253304_370.1","last_update":"2020-06-03T19:00:17.253304Z"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T19:02:43.505 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:02:43.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.507 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:43.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:43.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:43.812 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.828994_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.828994Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:43.812 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:02:43.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:43.814 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:43.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:43.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:44.117 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210962.828994_265.1","last_update":"2020-06-03T19:02:42.828994Z"},{"marker":"1_1591210558.304138_167.1","last_update":"2020-06-03T18:55:58.304138Z"},{"marker":"1_1591210817.253304_370.1","last_update":"2020-06-03T19:00:17.253304Z"},{"marker":"1_1591210517.328825_120.1","last_update":"2020-06-03T18:55:17.328825Z"}] 2020-06-03T19:02:44.118 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:02:44.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.119 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:44.439 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11569358039035221345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.828994_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.828994Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210558.304138_167.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:58.304138Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:44.439 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:02:44.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.441 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:44.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:44.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:44.763 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210962.861784_255.1","last_update":"2020-06-03T19:02:42.861784Z"},{"marker":"1_1591210580.323556_175.1","last_update":"2020-06-03T18:56:20.323556Z"},{"marker":"1_1591210817.251004_228.1","last_update":"2020-06-03T19:00:17.251004Z"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T19:02:44.763 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:02:44.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:44.765 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:44.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:44.810 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:45.045 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4441235937469265296},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.861784_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.861784Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.323556_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.323556Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.251004_228.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.251004Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:45.045 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:02:45.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.047 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:45.091 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:45.092 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:45.345 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210962.861784_255.1","last_update":"2020-06-03T19:02:42.861784Z"},{"marker":"1_1591210580.323556_175.1","last_update":"2020-06-03T18:56:20.323556Z"},{"marker":"1_1591210817.251004_228.1","last_update":"2020-06-03T19:00:17.251004Z"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T19:02:45.345 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:02:45.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.347 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:45.389 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:45.390 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:45.605 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6026847009502321472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.861784_255.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.861784Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.323556_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.323556Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.251004_228.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.251004Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:45.606 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:02:45.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.608 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:45.920 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210962.862478_240.1","last_update":"2020-06-03T19:02:42.862478Z"},{"marker":"1_1591210580.295126_132.1","last_update":"2020-06-03T18:56:20.295126Z"},{"marker":"1_1591210817.340370_180.1","last_update":"2020-06-03T19:00:17.340370Z"},{"marker":"1_1591210517.243048_104.1","last_update":"2020-06-03T18:55:17.243048Z"}] 2020-06-03T19:02:45.920 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:02:45.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:45.922 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:46.001 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:46.001 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:46.230 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210800.327592_227.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:00.327592Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.295126_132.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.295126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.340370_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.340370Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.243048_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.243048Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:46.231 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=0 master=1_1591210962.862478_240.1 target=1_1591210800.327592_227.6 2020-06-03T19:02:51.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.234 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:51.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:51.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:51.515 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.862478_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.862478Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.295126_132.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.295126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.340370_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.340370Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.243048_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.243048Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:51.516 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:02:51.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.518 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:51.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:51.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:51.820 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591210962.862478_240.1","last_update":"2020-06-03T19:02:42.862478Z"},{"marker":"1_1591210580.295126_132.1","last_update":"2020-06-03T18:56:20.295126Z"},{"marker":"1_1591210817.340370_180.1","last_update":"2020-06-03T19:00:17.340370Z"},{"marker":"1_1591210517.243048_104.1","last_update":"2020-06-03T18:55:17.243048Z"}] 2020-06-03T19:02:51.821 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:02:51.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:51.823 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:51.868 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:51.868 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:52.118 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":18141103562818399107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210962.862478_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.862478Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591210580.295126_132.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:56:20.295126Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.340370_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.340370Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.243048_104.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.243048Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:02:52.118 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:02:52.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:52.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:52.120 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:52.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:52.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:52.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:52.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:52.391 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:52.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:52.468 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:52.705 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:52.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:52.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:52.707 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:52.745 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:52.745 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:52.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:52.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:52.994 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:53.080 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:53.081 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:53.331 INFO:teuthology.orchestra.run.smithi061.stdout:[] 2020-06-03T19:02:53.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:53.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:53.333 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:53.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:53.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:53.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:53.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:02:53.608 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:02:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:53.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:53.930 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:02:53.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-06-03T19:02:53.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:53.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:53.932 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:53.972 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:53.972 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:54.002 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:02:54.001+0000 7f93eabd20c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:02:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:02:54.001+0000 7f93eabd20c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:02:54.007 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:02:54.005+0000 7f93eabd20c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:02:54.007 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:02:54.005+0000 7f93eabd20c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:02:54.022 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:02:54.023 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:54.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:02:54.024 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:02:54.105 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:02:54.105 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:02:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:02:54.447 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":2,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:02:54.448 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-06-03T19:03:54.449 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-49 2020-06-03T19:03:54.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:03:54.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:03:54.549 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:03:54.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:03:54.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:03:54.786 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:03:54.824 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211034.497528_172.1","last_update":"2020-06-03T19:03:54.497528Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:03:54.825 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:03:54.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:03:54.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:03:54.827 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:03:54.868 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:03:54.868 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:03:55.094 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:03:55.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:00.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:00.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:00.099 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:00.141 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:00.141 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:00.386 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:00.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:05.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:05.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:05.390 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:05.432 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:05.433 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:05.682 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:05.682 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:10.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:10.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:10.685 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:10.726 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:10.727 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:10.969 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:10.970 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:15.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:15.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:15.973 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:16.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:16.015 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:16.269 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:16.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:21.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:21.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:21.274 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:21.317 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:21.317 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:21.560 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:21.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:24.704 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:04:24.700+0000 7fc3a27f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:04:26.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:26.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:26.564 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:26.604 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:26.604 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:26.845 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:26.846 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:31.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:31.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:31.849 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:32.129 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:32.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:37.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:37.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:37.132 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:37.174 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:37.174 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:37.432 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:37.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:42.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:42.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:42.436 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:42.483 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:42.484 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:42.723 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:42.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:47.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:47.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:47.727 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:47.768 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:47.769 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:47.999 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:47.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:53.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:53.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:53.002 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:53.042 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:53.043 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:53.292 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591210962.545534_155.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:42.545534Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:53.293 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211034.497528_172.1 target=1_1591210962.545534_155.1 2020-06-03T19:04:54.699 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:04:54.694+0000 7fb2a5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:04:58.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.297 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:58.339 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:58.339 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:58.588 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:58.589 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:04:58.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:04:58.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.591 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:58.632 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:58.632 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:58.854 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:58.855 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:04:58.855 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:04:58.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:04:58.857 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:04:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:58.898 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:59.166 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:04:59.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:04:59.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=12983a69-4226-40f7-aaec-4a03acd75b02', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:04:59.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=12983a69-4226-40f7-aaec-4a03acd75b02', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:04:59.185 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=12983a69-4226-40f7-aaec-4a03acd75b02 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:04:59.246 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:59.246 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:59.277 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:04:59.275+0000 7f4f4503c0c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:04:59.278 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:04:59.275+0000 7f4f4503c0c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:04:59.279 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:04:59.275+0000 7f4f4503c0c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:04:59.279 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:04:59.275+0000 7f4f4503c0c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:04:59.293 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:04:59.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:04:59.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:04:59.296 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:04:59.381 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:04:59.381 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:04:59.640 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:04:59.731 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":3,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"12983a69-4226-40f7-aaec-4a03acd75b02"},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:04:59.732 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "12983a69-4226-40f7-aaec-4a03acd75b02" 2020-06-03T19:06:00.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:06:00.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:06:00.089 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:06:00.127 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:06:00.127 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:06:00.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:06:00.151+0000 7fbe40db70c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:06:00.156 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:06:00.151+0000 7fbe40db70c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:06:00.157 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:06:00.155+0000 7fbe40db70c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:06:00.157 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:06:00.155+0000 7fbe40db70c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:06:00.171 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:06:00.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:06:00.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:06:00.173 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:06:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:06:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:06:00.501 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:06:00.610 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":4,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"12983a69-4226-40f7-aaec-4a03acd75b02"},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:06:00.611 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-06-03T19:07:00.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:07:00.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:07:00.615 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:07:00.657 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:07:00.658 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:07:00.686 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:07:00.682+0000 7f21db7680c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:07:00.687 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:07:00.682+0000 7f21db7680c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:07:00.690 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:07:00.686+0000 7f21db7680c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:07:00.691 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:07:00.686+0000 7f21db7680c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:07:00.705 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:07:00.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:07:00.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:07:00.707 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:07:00.786 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:07:00.786 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:07:01.025 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:07:01.124 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":5,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:07:01.124 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-06-03T19:08:01.127 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-06-03T19:08:01.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.131 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:01.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:01.183 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:01.386 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:08:01.425 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211034.497528_172.1","last_update":"2020-06-03T19:03:54.497528Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:08:01.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:08:01.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.428 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:01.471 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:01.471 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:01.705 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:08:01.706 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:08:01.707 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:08:01.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.710 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:01.758 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:01.759 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:01.990 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:08:01.991 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:08:01.992 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:08:01.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:01.993 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:02.035 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:02.035 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:02.310 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211034.497528_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:03:54.497528Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:08:02.311 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:08:02.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:02.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'KH6AWYMVT2ZSL8JM', '--secret', 'PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:02.313 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key KH6AWYMVT2ZSL8JM --secret PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:02.393 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:02.393 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:02.461 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:02.461+0000 7f057e6cc0c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:08:02.461 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:02.461+0000 7f057e6cc0c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:08:02.463 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:02.461+0000 7f057e6cc0c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:08:02.464 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:02.461+0000 7f057e6cc0c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:08:02.479 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"70f73dee-ac21-43fe-afab-40c87360da22","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"KH6AWYMVT2ZSL8JM","secret_key":"PyftQEe4RgPHbgLSv5EF53IxoXfv9TAk"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:08:02.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:02.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:02.481 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:02.559 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:02.560 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:Sending period to new master zone 12983a69-4226-40f7-aaec-4a03acd75b02 2020-06-03T19:08:10.242 INFO:teuthology.orchestra.run.smithi061.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":6,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"70f73dee-ac21-43fe-afab-40c87360da22","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"70f73dee-ac21-43fe-afab-40c87360da22","val":4046703505},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:08:10.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.245 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:10.286 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:10.286 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.308 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:10.302+0000 7fbf92dd60c0 0 failed reading obj info from .rgw.root:zone_info.3ee2667c-4516-4846-aa22-24f7de69bd8e: (2) No such file or directory 2020-06-03T19:08:10.308 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:10.302+0000 7fbf92dd60c0 0 WARNING: could not read zone params for zone id=3ee2667c-4516-4846-aa22-24f7de69bd8e name=test-zone2 2020-06-03T19:08:10.309 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:10.306+0000 7fbf92dd60c0 0 failed reading obj info from .rgw.root:zone_info.b9904e04-437e-4cce-bbd1-9073303d0345: (2) No such file or directory 2020-06-03T19:08:10.309 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:10.306+0000 7fbf92dd60c0 0 WARNING: could not read zone params for zone id=b9904e04-437e-4cce-bbd1-9073303d0345 name=test-zone4 2020-06-03T19:08:10.321 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}} 2020-06-03T19:08:10.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.323 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:10.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:10.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.422 INFO:teuthology.orchestra.run.smithi112.stderr:zone name remove not found in zonegroup test-zonegroup 2020-06-03T19:08:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:08:10.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.433 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:10.476 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.517 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:10.517+0000 7fd5875240c0 0 failed reading obj info from .rgw.root:zone_info.12983a69-4226-40f7-aaec-4a03acd75b02: (2) No such file or directory 2020-06-03T19:08:10.517 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:10.517+0000 7fd5875240c0 0 WARNING: could not read zone params for zone id=12983a69-4226-40f7-aaec-4a03acd75b02 name=test-zone1 2020-06-03T19:08:10.518 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:10.517+0000 7fd5875240c0 0 failed reading obj info from .rgw.root:zone_info.a04bd580-0788-45da-8135-33bf41c24c11: (2) No such file or directory 2020-06-03T19:08:10.518 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:10.517+0000 7fd5875240c0 0 WARNING: could not read zone params for zone id=a04bd580-0788-45da-8135-33bf41c24c11 name=test-zone3 2020-06-03T19:08:10.522 INFO:teuthology.orchestra.run.smithi061.stderr:2020-06-03T19:08:10.521+0000 7fd5875240c0 0 zone id 70f73dee-ac21-43fe-afab-40c87360da22 is not a part of zonegroup default 2020-06-03T19:08:10.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.534 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.654 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:10.650+0000 7f807e0a30c0 0 WARNING: can't generate connection for zone 70f73dee-ac21-43fe-afab-40c87360da22 id remove: no endpoints defined 2020-06-03T19:08:10.851 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":7,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:08:10.852 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-06-03T19:08:10.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.854 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:10.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:10.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:10.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"0a0a70e3-d01b-4974-9059-4628bcb3b983","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}} 2020-06-03T19:08:10.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:10.962 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:11.101 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"68b71733-ff0a-48c8-9e3d-f2f8cb908c02","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4"} 2020-06-03T19:08:11.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.110 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.191 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:11.469 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":8,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"0a0a70e3-d01b-4974-9059-4628bcb3b983","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"68b71733-ff0a-48c8-9e3d-f2f8cb908c02","zones":[{"id":"68b71733-ff0a-48c8-9e3d-f2f8cb908c02","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"68b71733-ff0a-48c8-9e3d-f2f8cb908c02","val":3111998734},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:08:11.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.472 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.552 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:11.601 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:11.598+0000 7fd6ebae50c0 0 zone id 68b71733-ff0a-48c8-9e3d-f2f8cb908c02 is not a part of zonegroup test-zonegroup 2020-06-03T19:08:11.602 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:11.598+0000 7fd6ebae50c0 0 zone id 68b71733-ff0a-48c8-9e3d-f2f8cb908c02 is not a part of zonegroup default 2020-06-03T19:08:11.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.615 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.696 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:2020-06-03T19:08:11.730+0000 7eff66cc90c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-06-03T19:08:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-06-03T19:08:11.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.744 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:11.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:11.855 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:11.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:11.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:12.176 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","epoch":9,"predecessor_uuid":"c667fc06-9ec1-4c12-8825-c7c36661d280","sync_status":["","1_1591210885.066318_218.1","",""],"period_map":{"id":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","zones":[{"id":"12983a69-4226-40f7-aaec-4a03acd75b02","name":"test-zone1","endpoints":["http://smithi112.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"3ee2667c-4516-4846-aa22-24f7de69bd8e","name":"test-zone2","endpoints":["http://smithi061.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"a04bd580-0788-45da-8135-33bf41c24c11","name":"test-zone3","endpoints":["http://smithi112.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b9904e04-437e-4cce-bbd1-9073303d0345","name":"test-zone4","endpoints":["http://smithi061.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"12983a69-4226-40f7-aaec-4a03acd75b02","val":1338018264},{"key":"3ee2667c-4516-4846-aa22-24f7de69bd8e","val":1366013885},{"key":"a04bd580-0788-45da-8135-33bf41c24c11","val":2179005318},{"key":"b9904e04-437e-4cce-bbd1-9073303d0345","val":1649467902}]},"master_zonegroup":"test-zonegroup","master_zone":"12983a69-4226-40f7-aaec-4a03acd75b02","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"55979c05-ad33-4f8a-aebf-51e0a99d9db4","realm_name":"test-realm","realm_epoch":4} 2020-06-03T19:08:12.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-06-03T19:08:12.178 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-50 2020-06-03T19:08:12.989 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-51 2020-06-03T19:08:13.056 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-52 2020-06-03T19:08:13.121 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-53 2020-06-03T19:08:13.153 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-54 2020-06-03T19:08:13.214 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-55 2020-06-03T19:08:15.494 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-06-03T19:08:15.495 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-56 2020-06-03T19:08:15.536 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-57 2020-06-03T19:08:15.602 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-58 2020-06-03T19:08:15.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:15.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:15.662 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:16.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:16.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:16.015 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:16.098 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:16.099 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:16.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:16.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:08:16.416 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:08:16.498 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:16.498 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:16.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:16.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:16.801 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:16.876 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:16.877 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:17.123 INFO:teuthology.orchestra.run.smithi112.stdout:["bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","obj_delete_at_hint.0000000078","meta.history","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2-rhmmtn-43","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000026","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000028","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.27-rhmmtn-18","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.1-rhmmtn-50","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.31-rhmmtn-22","obj_delete_at_hint.0000000040","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","meta.full-sync.index.0","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.20-rhmmtn-11","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.32-rhmmtn-23","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","obj_delete_at_hint.0000000031","data_log.3","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.2","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.2","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.49-rhmmtn-40","obj_delete_at_hint.0000000110","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.1-rhmmtn-1","obj_delete_at_hint.0000000096","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.36-rhmmtn-27","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.5-rhmmtn-54","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.40-rhmmtn-31","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.2-rhmmtn-2","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.7-rhmmtn-56","obj_delete_at_hint.0000000041","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.39-rhmmtn-30","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","obj_delete_at_hint.0000000014","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.1","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.9-rhmmtn-58","obj_delete_at_hint.0000000074","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.23-rhmmtn-14","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1-rhmmtn-42","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.3-rhmmtn-46","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.50-rhmmtn-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.22-rhmmtn-13","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.16-rhmmtn-7","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7590.1-rhmmtn-49","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.2-rhmmtn-51","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.29-rhmmtn-20","obj_delete_at_hint.0000000065","datalog.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.4-rhmmtn-47","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","mdlog.sync-status.shard.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.4-rhmmtn-53","obj_delete_at_hint.0000000116","datalog.sync-status.a04bd580-0788-45da-8135-33bf41c24c11","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.2-rhmmtn-45","obj_delete_at_hint.0000000034","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","obj_delete_at_hint.0000000101","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.0","mdlog.sync-status.shard.1","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.41-rhmmtn-32","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.1","obj_delete_at_hint.0000000066","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.19-rhmmtn-10","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.6-rhmmtn-6","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.47-rhmmtn-38","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.3","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.2","mdlog.sync-status.shard.0","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.8-rhmmtn-57","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","obj_delete_at_hint.0000000039","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.46-rhmmtn-37","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.37-rhmmtn-28","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.3","obj_delete_at_hint.0000000100","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","obj_delete_at_hint.0000000093","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.0","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.25-rhmmtn-16","obj_delete_at_hint.0000000012","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.6-rhmmtn-55","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.42-rhmmtn-33","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.43-rhmmtn-34","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.3-rhmmtn-52","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","obj_delete_at_hint.0000000072","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.5-rhmmtn-48","obj_delete_at_hint.0000000062","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.0","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.44-rhmmtn-35","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.1-rhmmtn-44","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.3-rhmmtn-3","obj_delete_at_hint.0000000108","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","obj_delete_at_hint.0000000094","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.30-rhmmtn-21","obj_delete_at_hint.0000000060","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.48-rhmmtn-39","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","obj_delete_at_hint.0000000051","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.28-rhmmtn-19","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","obj_delete_at_hint.0000000023","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.5-rhmmtn-5","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.3","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","obj_delete_at_hint.0000000086","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.26-rhmmtn-17","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.1","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.24-rhmmtn-15","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.3","meta.log.b2e295e4-c2ab-4a12-b669-6be057e86c4f.1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.21-rhmmtn-12","obj_delete_at_hint.0000000124","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.1","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.45-rhmmtn-36","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","obj_delete_at_hint.0000000097","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.17-rhmmtn-8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.38-rhmmtn-29","obj_delete_at_hint.0000000042","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.4-rhmmtn-4","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.18-rhmmtn-9","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-06-03T19:08:17.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.155 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:17.454 INFO:teuthology.orchestra.run.smithi061.stdout:["bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.1","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2-rhmmtn-43","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000026","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000028","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.27-rhmmtn-18","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.31-rhmmtn-22","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.20-rhmmtn-11","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.32-rhmmtn-23","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","obj_delete_at_hint.0000000024","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","obj_delete_at_hint.0000000000","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","obj_delete_at_hint.0000000031","data_log.3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","obj_delete_at_hint.0000000005","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.49-rhmmtn-40","obj_delete_at_hint.0000000110","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.1-rhmmtn-1","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.3","obj_delete_at_hint.0000000096","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.5-rhmmtn-54","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.40-rhmmtn-31","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.2-rhmmtn-2","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.0","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.23-rhmmtn-14","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1-rhmmtn-42","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.3-rhmmtn-46","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.50-rhmmtn-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.22-rhmmtn-13","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.16-rhmmtn-7","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7590.1-rhmmtn-49","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.2-rhmmtn-51","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.29-rhmmtn-20","obj_delete_at_hint.0000000065","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.4-rhmmtn-47","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","mdlog.sync-status.shard.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","obj_delete_at_hint.0000000116","datalog.sync-status.a04bd580-0788-45da-8135-33bf41c24c11","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.2-rhmmtn-45","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.0","mdlog.sync-status.shard.1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.1","obj_delete_at_hint.0000000066","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.19-rhmmtn-10","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.6-rhmmtn-6","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.47-rhmmtn-38","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.3","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.2","mdlog.sync-status.shard.0","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.8-rhmmtn-57","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","obj_delete_at_hint.0000000039","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.46-rhmmtn-37","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.37-rhmmtn-28","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.3","obj_delete_at_hint.0000000100","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000093","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.25-rhmmtn-16","obj_delete_at_hint.0000000012","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.43-rhmmtn-34","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.0","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","obj_delete_at_hint.0000000072","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.5-rhmmtn-48","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","obj_delete_at_hint.0000000062","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.0","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.1-rhmmtn-44","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.3-rhmmtn-3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.30-rhmmtn-21","obj_delete_at_hint.0000000060","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.48-rhmmtn-39","obj_delete_at_hint.0000000109","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","obj_delete_at_hint.0000000013","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","data.full-sync.index.a04bd580-0788-45da-8135-33bf41c24c11.2","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.28-rhmmtn-19","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","obj_delete_at_hint.0000000023","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.5-rhmmtn-5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","data_log.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.26-rhmmtn-17","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.24-rhmmtn-15","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","meta.log.b2e295e4-c2ab-4a12-b669-6be057e86c4f.1","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.21-rhmmtn-12","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","obj_delete_at_hint.0000000124","datalog.sync-status.shard.a04bd580-0788-45da-8135-33bf41c24c11.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","bucket.sync-status.a04bd580-0788-45da-8135-33bf41c24c11:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.45-rhmmtn-36","obj_delete_at_hint.0000000097","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.17-rhmmtn-8","obj_delete_at_hint.0000000042","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.4-rhmmtn-4","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.18-rhmmtn-9","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-06-03T19:08:17.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.488 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:17.764 INFO:teuthology.orchestra.run.smithi112.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.1","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.2-rhmmtn-43","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000028","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.27-rhmmtn-18","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.31-rhmmtn-22","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000040","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","meta.full-sync.index.0","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.20-rhmmtn-11","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.32-rhmmtn-23","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","obj_delete_at_hint.0000000024","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.2","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.2","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.2","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.49-rhmmtn-40","obj_delete_at_hint.0000000110","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.1-rhmmtn-1","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.3","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.2-rhmmtn-2","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.0","obj_delete_at_hint.0000000014","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.1","obj_delete_at_hint.0000000064","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","obj_delete_at_hint.0000000071","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.9-rhmmtn-58","obj_delete_at_hint.0000000074","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.23-rhmmtn-14","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2020-06-03-19-3ee2667c-4516-4846-aa22-24f7de69bd8e.6472.1-rhmmtn-42","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.3-rhmmtn-46","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.50-rhmmtn-41","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.22-rhmmtn-13","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.16-rhmmtn-7","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","meta.log.c667fc06-9ec1-4c12-8825-c7c36661d280.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7590.1-rhmmtn-49","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.29-rhmmtn-20","obj_delete_at_hint.0000000065","datalog.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.4-rhmmtn-47","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","mdlog.sync-status.shard.2","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","obj_delete_at_hint.0000000116","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.2-rhmmtn-45","obj_delete_at_hint.0000000034","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.41-rhmmtn-32","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","obj_delete_at_hint.0000000066","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.19-rhmmtn-10","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.6-rhmmtn-6","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.47-rhmmtn-38","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.2","obj_delete_at_hint.0000000039","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.46-rhmmtn-37","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","obj_delete_at_hint.0000000100","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.0","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","obj_delete_at_hint.0000000057","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.25-rhmmtn-16","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.6-rhmmtn-55","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","meta.log.f641ddd2-34e8-45d4-a5d7-f7dfb77827d9.0","obj_delete_at_hint.0000000020","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.0","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7833.3-rhmmtn-52","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","obj_delete_at_hint.0000000072","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.5-rhmmtn-48","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.44-rhmmtn-35","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.7269.1-rhmmtn-44","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.3-rhmmtn-3","obj_delete_at_hint.0000000108","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","obj_delete_at_hint.0000000094","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.30-rhmmtn-21","obj_delete_at_hint.0000000060","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.48-rhmmtn-39","obj_delete_at_hint.0000000109","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","obj_delete_at_hint.0000000013","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.35-rhmmtn-26","obj_delete_at_hint.0000000051","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.28-rhmmtn-19","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","obj_delete_at_hint.0000000023","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.34-rhmmtn-25","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.5-rhmmtn-5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","data_log.2","obj_delete_at_hint.0000000067","meta.log.f641ddd2-34e8-45d4-a5d7-f7dfb77827d9.1","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.3","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","obj_delete_at_hint.0000000086","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.26-rhmmtn-17","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","data.full-sync.index.3ee2667c-4516-4846-aa22-24f7de69bd8e.1","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.33-rhmmtn-24","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.24-rhmmtn-15","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.3ee2667c-4516-4846-aa22-24f7de69bd8e.3","meta.log.b2e295e4-c2ab-4a12-b669-6be057e86c4f.1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.21-rhmmtn-12","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","obj_delete_at_hint.0000000124","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.45-rhmmtn-36","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","obj_delete_at_hint.0000000097","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.17-rhmmtn-8","2020-06-03-19-12983a69-4226-40f7-aaec-4a03acd75b02.4472.38-rhmmtn-29","obj_delete_at_hint.0000000042","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.4-rhmmtn-4","2020-06-03-18-12983a69-4226-40f7-aaec-4a03acd75b02.4472.18-rhmmtn-9","bucket.sync-status.3ee2667c-4516-4846-aa22-24f7de69bd8e:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-06-03T19:08:17.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:17.796 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:17.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:17.832 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:18.072 INFO:teuthology.orchestra.run.smithi061.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-46:12983a69-4226-40f7-aaec-4a03acd75b02.7269.3:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-14:12983a69-4226-40f7-aaec-4a03acd75b02.4472.23:1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:5","obj_delete_at_hint.0000000024","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:3","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:5","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.2","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.3","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:3","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.0","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-13:12983a69-4226-40f7-aaec-4a03acd75b02.4472.22:1","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","meta.log.c667fc06-9ec1-4c12-8825-c7c36661d280.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-25:12983a69-4226-40f7-aaec-4a03acd75b02.4472.34:8","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-20:12983a69-4226-40f7-aaec-4a03acd75b02.4472.29:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-49:12983a69-4226-40f7-aaec-4a03acd75b02.7590.1:5","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-47:12983a69-4226-40f7-aaec-4a03acd75b02.7269.4:8","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-37:12983a69-4226-40f7-aaec-4a03acd75b02.4472.46:10","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:1","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","datalog.sync-status.shard.12983a69-4226-40f7-aaec-4a03acd75b02.2","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:5","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:1","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-24:12983a69-4226-40f7-aaec-4a03acd75b02.4472.33:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-23:12983a69-4226-40f7-aaec-4a03acd75b02.4472.32:8","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-48:12983a69-4226-40f7-aaec-4a03acd75b02.7269.5:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","meta.log.f641ddd2-34e8-45d4-a5d7-f7dfb77827d9.0","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","data.full-sync.index.12983a69-4226-40f7-aaec-4a03acd75b02.0","obj_delete_at_hint.0000000072","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-26:12983a69-4226-40f7-aaec-4a03acd75b02.4472.35:8","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:5","obj_delete_at_hint.0000000013","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-36:12983a69-4226-40f7-aaec-4a03acd75b02.4472.45:10","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-15:12983a69-4226-40f7-aaec-4a03acd75b02.4472.24:1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-12:12983a69-4226-40f7-aaec-4a03acd75b02.4472.21:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-18:12983a69-4226-40f7-aaec-4a03acd75b02.4472.27:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-10:12983a69-4226-40f7-aaec-4a03acd75b02.4472.19:3","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-16:12983a69-4226-40f7-aaec-4a03acd75b02.4472.25:1","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-19:12983a69-4226-40f7-aaec-4a03acd75b02.4472.28:8","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-21:12983a69-4226-40f7-aaec-4a03acd75b02.4472.30:8","obj_delete_at_hint.0000000067","meta.log.f641ddd2-34e8-45d4-a5d7-f7dfb77827d9.1","obj_delete_at_hint.0000000086","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:7","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-22:12983a69-4226-40f7-aaec-4a03acd75b02.4472.31:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","meta.log.b2e295e4-c2ab-4a12-b669-6be057e86c4f.1","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-38:12983a69-4226-40f7-aaec-4a03acd75b02.4472.47:10","obj_delete_at_hint.0000000124","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-17:12983a69-4226-40f7-aaec-4a03acd75b02.4472.26:1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-11:12983a69-4226-40f7-aaec-4a03acd75b02.4472.20:1","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-06-03T19:08:18.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:18.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:18.207 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:18.248 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:18.248 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:18.489 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211296.765961_360.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_527.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_478.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_415.2","last_update":"2020-06-03T19:08:16.765961Z"}] 2020-06-03T19:08:18.489 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:08:18.490 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:18.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:18.491 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:18.532 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:18.533 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:18.787 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:18.788 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:23.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:23.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:23.791 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:23.836 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:23.837 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:24.080 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:24.081 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:29.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:29.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:29.085 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:29.130 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:29.130 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:29.385 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:29.386 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:34.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:34.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:34.390 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:34.435 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:34.435 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:34.689 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:34.690 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:39.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:39.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:39.693 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:39.736 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:39.736 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:39.992 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:39.993 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:44.683 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc39ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.684 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc39efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.684 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3b5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.684 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3b4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.687 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3a1ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.688 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc39d7ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.688 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3a77fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.689 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3a67fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.689 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.680+0000 7fc3a57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.689 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.684+0000 7fc39efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.689 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.684+0000 7fc3b5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.692 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a07f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.692 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a27f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.692 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39cfe9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.692 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39b7e6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.692 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3b57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.693 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3b6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.693 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3b77fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.693 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a7fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.693 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39bfe7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.693 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39c7e8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.694 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.694 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.694 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a47f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.694 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a3ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.694 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3b4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a1ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc3a17f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.695 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.696 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:08:44.688+0000 7fc39dfeb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:08:44.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:44.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:44.995 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:45.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:45.033 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:45.282 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:45.283 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:50.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:50.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:50.286 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:50.582 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:50.583 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:08:55.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:55.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:08:55.586 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:08:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:08:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:08:55.877 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:08:55.877 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:00.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:00.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:00.881 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:00.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:00.935 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:01.186 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:01.187 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:06.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:06.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:06.189 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:06.232 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:06.233 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:06.500 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:06.501 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:11.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:11.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:11.503 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:11.548 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:11.548 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:11.800 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:11.801 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:14.027 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc58ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.027 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc50ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.027 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc51ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc52ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc597fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc4bfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc4b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc48ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.028 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc59ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.029 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc3fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.029 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc42ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.029 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc3d7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.029 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc40ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.029 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc417fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.030 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc4a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.030 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc407f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.030 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc4affd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.030 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc3cff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.030 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc41ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.031 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.024+0000 7fcc497fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.032 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.032+0000 7fcc58ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:14.033 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:09:14.032+0000 7fcc50ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:09:16.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:16.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:16.804 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:16.852 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:16.852 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:17.121 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_360.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_527.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_478.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_415.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:17.122 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:09:17.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:17.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:17.124 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:17.397 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211296.765961_360.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_527.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_478.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_415.2","last_update":"2020-06-03T19:08:16.765961Z"}] 2020-06-03T19:09:17.398 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:09:17.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:17.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:17.400 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:17.483 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:17.483 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:17.708 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:17.709 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:22.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:22.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:22.718 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:22.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:22.764 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:22.996 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:22.997 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:27.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:27.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:27.999 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:28.046 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:28.046 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:28.279 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:28.280 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:33.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:33.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:33.282 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:33.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:33.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:33.568 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:33.569 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:38.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:38.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:38.572 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:38.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:38.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:38.853 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:38.854 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:43.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:43.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:43.857 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:43.899 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:43.899 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:44.130 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:44.131 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:49.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:49.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:49.134 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:49.176 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:49.410 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:49.410 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:54.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:54.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:54.419 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:54.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:54.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:54.672 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:54.673 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:09:59.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:59.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:09:59.676 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:09:59.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:09:59.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:09:59.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:09:59.960 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:10:04.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:04.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:04.962 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:05.004 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:05.005 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:05.251 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:05.251 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:10:10.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:10.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:10.255 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:10.298 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:10.299 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:10.530 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.575715_266.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.575715Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211160.564127_406.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:00.564127Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.253304_370.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.253304Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.328825_120.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.328825Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:10.530 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591211296.765961_360.3 target=1_1591210966.575715_266.1, shard=1 master=1_1591211296.765961_527.3 target=1_1591211160.564127_406.1, shard=2 master=1_1591211296.765961_478.3 target=1_1591210817.253304_370.1, shard=3 master=1_1591211296.765961_415.2 target=1_1591210517.328825_120.1 2020-06-03T19:10:14.013 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.011+0000 7fb2c5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.013 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.011+0000 7fb2c57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.015 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.011+0000 7fb29aff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.015 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.011+0000 7fb2997f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.018 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb2bd7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.018 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb2c4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.018 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb29ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.019 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb2bf7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.019 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb299ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:14.019 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:14.015+0000 7fb2beffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:15.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:15.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:15.534 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:15.575 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:15.576 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:15.805 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_360.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_527.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_478.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_415.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:15.805 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:10:15.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:15.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:15.807 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:15.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:15.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:16.102 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211296.765961_360.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_527.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_478.3","last_update":"2020-06-03T19:08:16.765961Z"},{"marker":"1_1591211296.765961_415.2","last_update":"2020-06-03T19:08:16.765961Z"}] 2020-06-03T19:10:16.103 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:10:16.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.105 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:16.422 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11569358039035221345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_360.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_527.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_478.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211296.765961_415.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:08:16.765961Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:16.423 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:10:16.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.425 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:16.466 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:16.466 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:16.727 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210966.588355_256.1","last_update":"2020-06-03T19:02:46.588355Z"},{"marker":"1_1591211182.752489_408.1","last_update":"2020-06-03T19:06:22.752489Z"},{"marker":"1_1591210817.251004_228.1","last_update":"2020-06-03T19:00:17.251004Z"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T19:10:16.727 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:10:16.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:16.729 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:16.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:16.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:17.015 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4441235937469265296},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.588355_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.588355Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211182.752489_408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:22.752489Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.251004_228.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.251004Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:17.015 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:10:17.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.017 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:17.308 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591210966.588355_256.1","last_update":"2020-06-03T19:02:46.588355Z"},{"marker":"1_1591211182.752489_408.1","last_update":"2020-06-03T19:06:22.752489Z"},{"marker":"1_1591210817.251004_228.1","last_update":"2020-06-03T19:00:17.251004Z"},{"marker":"1_1591210517.148277_123.1","last_update":"2020-06-03T18:55:17.148277Z"}] 2020-06-03T19:10:17.308 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:10:17.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.310 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:17.353 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:17.353 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:17.571 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6026847009502321472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591210966.588355_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:02:46.588355Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211182.752489_408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:06:22.752489Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591210817.251004_228.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:00:17.251004Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591210517.148277_123.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T18:55:17.148277Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:17.571 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:10:17.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.573 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:17.658 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:17.852 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211417.699474_315.1","last_update":"2020-06-03T19:10:17.699474Z"},{"marker":"1_1591211417.702753_394.1","last_update":"2020-06-03T19:10:17.702753Z"},{"marker":"1_1591211417.705983_257.1","last_update":"2020-06-03T19:10:17.705983Z"},{"marker":"1_1591211417.691660_315.1","last_update":"2020-06-03T19:10:17.691660Z"}] 2020-06-03T19:10:17.852 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:10:17.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:17.854 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:17.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:17.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:18.114 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211417.699474_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.699474Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.702753_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.702753Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211417.705983_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.705983Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211417.691660_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.691660Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:18.115 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:10:18.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:18.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:18.116 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:18.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:18.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:18.410 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211417.699474_315.1","last_update":"2020-06-03T19:10:17.699474Z"},{"marker":"1_1591211417.702753_394.1","last_update":"2020-06-03T19:10:17.702753Z"},{"marker":"1_1591211417.705983_257.1","last_update":"2020-06-03T19:10:17.705983Z"},{"marker":"1_1591211417.691660_315.1","last_update":"2020-06-03T19:10:17.691660Z"}] 2020-06-03T19:10:18.411 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:10:18.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:18.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:18.412 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:18.456 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:18.456 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:18.719 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":18141103562818399107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211417.699474_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.699474Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.702753_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.702753Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211417.705983_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.705983Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211417.691660_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.691660Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:10:18.720 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:10:18.720 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-06-03T19:10:18.721 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-59 2020-06-03T19:10:18.766 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-60 2020-06-03T19:10:18.836 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-61 2020-06-03T19:10:19.205 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:19.203+0000 7fb29effd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:19.205 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:19.203+0000 7fb29b7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:19.206 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:10:19.203+0000 7fb2a77fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:19.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:19.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:19.471 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:19.483 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:10:19.478+0000 7fcc3e7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:10:19.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:19.530 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:19.745 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:19.745 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:19.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:19.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:19.747 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:19.793 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:19.793 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:19.976 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:19.978 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:20.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.002 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:20.090 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:20.090 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:20.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.341 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:20.391 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:20.391 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:20.602 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:20.602 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:20.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.640 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:20.717 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:20.717 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:20.908 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:20.908 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:20.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:20.949 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:21.000 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:21.000 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:21.205 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:21.205 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:21.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.241 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:21.292 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:21.293 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:21.506 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:21.507 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:21.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.568 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:21.623 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:21.623 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:21.813 INFO:teuthology.orchestra.run.smithi061.stderr:could not get bucket info for bucket=rhmmtn-59 2020-06-03T19:10:21.813 INFO:teuthology.orchestra.run.smithi061.stderr:ERROR: could not init bucket: (2) No such file or directory 2020-06-03T19:10:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-03T19:10:21.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:21.846 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:21.899 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:21.899 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:22.162 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:19.298337Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:10:19.058395Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:19.058431Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:10:19.298371Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:22.163 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:22.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.165 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:22.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:22.206 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:22.441 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:22.442 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:22.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.443 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:22.528 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:22.528 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:22.772 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:19.258467Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:10:19.058479Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:19.058453Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:10:19.258495Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:22.773 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:22.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:22.775 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:22.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:22.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:23.096 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:23.097 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:23.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.099 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:23.159 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:23.160 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:23.439 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:18.999650Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:10:18.950233Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:18.901701Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:10:19.048327Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:23.440 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:23.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.442 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:23.542 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:23.543 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:23.798 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.28.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.96.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:23.798 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:10:23.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:23.800 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:23.847 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:23.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:24.113 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-06-03T19:10:19.658518Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:10:19.254336Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5","timestamp":"2020-06-03T19:10:19.258465Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.96.5","timestamp":"2020-06-03T19:10:19.654310Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:24.114 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:10:24.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.116 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:24.161 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:24.161 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:24.430 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.28.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.96.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:24.431 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:10:24.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.432 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:24.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:24.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:24.715 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-06-03T19:10:19.658361Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:10:19.258495Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5","timestamp":"2020-06-03T19:10:19.258467Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.96.5","timestamp":"2020-06-03T19:10:19.654313Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:24.716 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:10:24.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:24.718 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:25.029 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.75.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.28.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:25.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:10:25.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.031 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:25.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:25.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:25.375 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-06-03T19:10:19.538389Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:10:19.331063Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5","timestamp":"2020-06-03T19:10:19.330500Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.495869Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:25.375 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:10:25.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.377 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:25.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:25.694 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.75.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.28.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:25.694 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:10:25.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:25.696 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:25.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:25.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:25.999 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-06-03T19:10:19.578271Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:10:19.334352Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5","timestamp":"2020-06-03T19:10:19.334323Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.530251Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:25.999 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:10:26.000 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone2, test-zone1} 2020-06-03T19:10:26.011 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:26.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:26.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:26.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:26.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:26.020 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:26.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:26.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:26.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:26.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:26.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:26.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:26.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:26.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:26.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:26.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:26.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:26.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:26.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:26.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:26.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:27.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:27.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone2, test-zone1} 2020-06-03T19:10:27.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone3, test-zone1} 2020-06-03T19:10:27.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:27.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:27.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:27.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:27.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:27.184 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:27.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:27.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:27.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:27.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:27.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:27.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:27.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:27.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:27.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:27.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:27.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:27.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:27.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:28.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:28.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:28.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:28.331 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone3, test-zone1} 2020-06-03T19:10:28.332 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone3, test-zone2} 2020-06-03T19:10:28.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:28.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:28.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:28.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:28.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:28.352 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:28.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:28.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:28.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:28.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:28.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:28.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:28.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:28.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:28.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:28.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:28.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:28.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:29.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:29.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:29.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:29.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:29.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone3, test-zone2} 2020-06-03T19:10:29.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:29.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:29.475 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:29.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:29.518 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:29.758 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:29.759 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:29.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:29.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:29.761 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:29.806 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:29.807 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:30.065 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.454411Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-03T19:10:19.454369Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.413984Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:19.813036Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:30.066 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:30.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.068 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:30.111 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:30.111 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:30.349 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:30.349 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:30.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.351 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:30.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:30.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:30.682 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.498384Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-03T19:10:19.498362Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.456430Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:19.862416Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:30.682 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:30.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.684 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:30.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:30.972 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:30.972 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:30.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:30.974 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:31.018 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:31.019 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:31.291 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.189019Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-03T19:10:19.142132Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.093873Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-03T19:10:19.237542Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:31.291 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:10:31.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.293 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:31.376 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:31.376 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:31.619 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.89.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:31.620 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:10:31.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.622 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:31.921 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-03T19:10:19.216051Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-03T19:10:19.216034Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:10:19.215632Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.498345Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:31.922 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:10:31.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:31.924 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:31.964 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:32.205 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.89.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:32.206 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:10:32.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.208 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:32.252 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:32.253 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:32.499 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-03T19:10:19.239244Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-03T19:10:19.241550Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:10:19.286459Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.498358Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:32.500 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:10:32.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.502 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:32.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:32.550 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:32.770 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:32.821 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:10:32.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:32.823 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:32.875 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:32.875 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:33.142 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:10:19.542257Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-03T19:10:19.499555Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:10:19.503992Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.778400Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:33.142 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:10:33.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:33.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:33.144 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:33.466 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:33.466 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:10:33.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:33.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:33.468 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:33.515 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:33.516 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:33.794 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:10:19.536308Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-03T19:10:19.536495Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:10:19.537691Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-03T19:10:19.733759Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:33.794 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:10:33.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone2, test-zone1} 2020-06-03T19:10:33.806 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:33.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:33.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:33.814 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:33.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:33.815 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:33.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:33.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:33.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:33.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:33.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:33.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:34.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:34.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:34.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:34.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:34.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:34.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:34.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:34.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:35.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:35.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:35.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone2, test-zone1} 2020-06-03T19:10:35.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone3, test-zone1} 2020-06-03T19:10:35.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:35.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:35.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:35.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:35.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:35.272 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:35.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:35.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:35.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:35.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:35.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:35.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:35.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:35.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:35.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:35.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:35.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:35.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:36.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:36.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:36.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:36.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:36.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone3, test-zone1} 2020-06-03T19:10:36.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone3, test-zone2} 2020-06-03T19:10:36.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:36.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:36.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:36.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:36.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:36.465 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:36.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:36.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:36.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:36.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:36.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:36.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:36.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:36.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:36.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:36.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:37.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:37.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:37.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:37.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:37.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:37.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:37.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone3, test-zone2} 2020-06-03T19:10:37.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:37.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:37.628 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:37.908 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:37.909 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:37.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:37.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:37.910 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:37.954 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:37.955 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:38.218 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:19.862532Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-03T19:10:19.862614Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:10:19.824257Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:10:19.858453Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:38.218 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:38.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.220 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:38.263 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:38.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:38.515 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:38.516 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:38.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.519 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:38.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:38.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:38.849 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:19.822478Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-03T19:10:19.862446Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:10:19.824771Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:10:19.866303Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:38.850 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:38.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:38.853 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:38.933 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:38.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:39.170 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:39.170 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:39.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.173 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:39.496 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5","timestamp":"2020-06-03T19:10:19.380987Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-03T19:10:19.334574Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:10:19.287997Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:10:19.431370Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:39.497 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:10:39.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.500 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:39.581 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:39.822 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.61.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:39.822 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:10:39.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:39.824 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:39.870 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:40.121 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T19:10:19.853285Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.894304Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-03T19:10:19.898279Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:10:19.858310Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:40.122 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:10:40.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.125 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:40.169 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:40.169 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:40.425 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.61.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:40.426 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:10:40.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.427 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:40.478 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:40.479 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:40.711 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T19:10:19.858405Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:10:19.862381Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-03T19:10:19.858429Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:10:19.858436Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:40.712 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:10:40.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:40.714 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:40.795 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:40.795 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:41.029 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.46.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:41.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:10:41.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.032 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:41.120 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:41.390 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-03T19:10:19.497993Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:10:19.538338Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T19:10:19.534295Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:10:19.538371Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:41.390 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:10:41.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.392 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:41.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:41.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:41.712 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.46.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:10:41.712 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:10:41.713 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:41.714 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:41.758 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:41.758 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:42.025 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-03T19:10:19.578309Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:10:19.578322Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-03T19:10:19.536550Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:10:19.507689Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:10:42.026 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:10:42.027 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone2, test-zone1} 2020-06-03T19:10:42.040 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:42.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:42.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:42.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:42.050 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:42.050 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:42.059 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:42.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:42.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:42.061 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:42.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:42.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:42.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:42.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:42.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:42.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:42.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:42.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:42.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:42.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:42.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:43.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:43.221 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone2, test-zone1} 2020-06-03T19:10:43.222 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone3, test-zone1} 2020-06-03T19:10:43.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:43.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:43.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:43.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:43.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:43.239 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:43.246 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:43.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:43.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:43.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:43.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:43.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:43.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:43.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:43.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:43.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:43.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:43.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:44.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:44.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:44.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:44.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:44.431 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone3, test-zone1} 2020-06-03T19:10:44.432 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone3, test-zone2} 2020-06-03T19:10:44.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:10:44.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:44.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:44.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:44.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:44.452 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:10:44.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:10:44.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:10:44.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:10:44.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:10:44.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:44.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:44.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:10:44.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:44.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:44.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:10:45.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:45.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:45.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:10:45.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:45.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:45.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:10:45.630 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone3, test-zone2} 2020-06-03T19:10:45.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:45.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:45.632 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:45.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:45.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:45.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:45.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:45.992 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:46.072 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:46.073 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:46.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:46.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:46.401 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:46.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:46.494 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:46.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:46.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:46.806 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:46.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:46.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:47.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:47.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:47.203 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:47.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:47.285 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:47.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:47.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:10:47.614 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:10:47.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:47.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:48.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:48.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:48.692 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:48.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:48.736 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:48.982 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.52.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.91.5"},{"key":3,"val":"00000000004.182.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.46.1"},{"key":6,"val":"00000000004.69.1"},{"key":7,"val":"00000000006.75.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.39.1"},{"key":10,"val":"00000000004.45.1"}]} 2020-06-03T19:10:48.982 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:48.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:48.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:48.984 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:49.030 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:49.030 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:49.303 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.330474Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.326379Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:48.289691Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.350805Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:48.291881Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.350795Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:48.289651Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:48.276967Z"}}}] 2020-06-03T19:10:49.304 WARNING:rgw_multi.tests:bucket rhmmtn-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.46.1 target=, shard=9 master=00000000004.39.1 target= 2020-06-03T19:10:54.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.307 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:54.627 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.330474Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-03T19:10:54.098465Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.326379Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:48.289691Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.350805Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:54.098436Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:48.291881Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.350795Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:48.289651Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-03T19:10:54.098473Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:48.276967Z"}}}] 2020-06-03T19:10:54.628 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:54.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.630 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:54.671 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:54.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:54.916 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.52.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.91.5"},{"key":3,"val":"00000000004.182.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.46.1"},{"key":6,"val":"00000000004.69.1"},{"key":7,"val":"00000000006.75.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.39.1"},{"key":10,"val":"00000000004.45.1"}]} 2020-06-03T19:10:54.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:54.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:54.921 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:55.002 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:55.003 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:55.257 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.268145Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-03T19:10:54.087053Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.294659Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:48.260447Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.350402Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:54.087036Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:48.294639Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.331037Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:48.305613Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-03T19:10:54.130280Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:48.294651Z"}}}] 2020-06-03T19:10:55.258 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:55.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:55.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:55.260 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:55.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:55.345 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:55.587 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.52.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.91.5"},{"key":3,"val":"00000000004.182.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.46.1"},{"key":6,"val":"00000000004.69.1"},{"key":7,"val":"00000000006.75.5"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.39.1"},{"key":10,"val":"00000000004.45.1"}]} 2020-06-03T19:10:55.588 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:10:55.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:55.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:10:55.590 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:10:55.634 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:10:55.635 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:10:55.898 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:46.354392Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:46.357459Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:46.359028Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:46.355712Z"}}}] 2020-06-03T19:10:55.899 WARNING:rgw_multi.tests:bucket rhmmtn-59 zone test-zone4 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.46.1 target=, shard=9 master=00000000004.39.1 target= 2020-06-03T19:11:00.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:00.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:00.902 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:00.946 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:00.947 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:01.223 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:46.354392Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:46.357459Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:46.359028Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:46.355712Z"}}}] 2020-06-03T19:11:01.224 WARNING:rgw_multi.tests:bucket rhmmtn-59 zone test-zone4 behind zone test-zone1: shard=1 master=00000000004.48.1 target=, shard=5 master=00000000004.46.1 target=, shard=9 master=00000000004.39.1 target= 2020-06-03T19:11:06.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.227 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:06.271 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:06.272 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:06.544 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.52.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-03T19:10:46.354668Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.91.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.182.1","timestamp":"2020-06-03T19:10:46.354392Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:46.355796Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-03T19:10:46.357459Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.75.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:10:46.359028Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-03T19:10:46.354284Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1","timestamp":"2020-06-03T19:10:46.355712Z"}}}] 2020-06-03T19:11:06.545 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-59 2020-06-03T19:11:06.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.546 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:06.628 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:06.628 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:06.884 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.31.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.98.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:06.885 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:11:06.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:06.888 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:06.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:06.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:07.193 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5","timestamp":"2020-06-03T19:10:19.658518Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:10:19.254336Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5","timestamp":"2020-06-03T19:10:19.258465Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.96.5","timestamp":"2020-06-03T19:10:19.654310Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:07.194 WARNING:rgw_multi.tests:bucket rhmmtn-59 zone test-zone1 behind zone test-zone2: shard=0 master=00000000002.37.5 target=00000000001.34.5, shard=2 master=00000000002.76.5 target=00000000001.74.5, shard=4 master=00000000002.31.5 target=00000000001.28.5, shard=7 master=00000000002.98.5 target=00000000001.96.5 2020-06-03T19:11:12.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.201 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:12.249 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:12.249 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:12.509 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.37.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.98.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:12.509 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:11:12.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.511 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:12.557 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:12.797 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.31.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.98.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:12.797 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:11:12.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:12.799 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:12.842 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:13.077 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.37.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.98.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:13.078 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-59 2020-06-03T19:11:13.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.080 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:13.161 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:13.398 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.66.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:13.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:11:13.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.400 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:13.741 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.36.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:13.742 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:11:13.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:13.745 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:13.823 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:14.058 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.66.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:14.059 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:11:14.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:14.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:14.061 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:14.107 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:14.107 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:14.362 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.36.5","timestamp":"2020-06-03T19:10:48.131241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.083990Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5","timestamp":"2020-06-03T19:10:48.035393Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.66.5","timestamp":"2020-06-03T19:10:48.183247Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:14.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-59 2020-06-03T19:11:14.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone2, test-zone1} 2020-06-03T19:11:14.379 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:14.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:14.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:14.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:14.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:14.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:14.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:14.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:14.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:14.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:14.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:14.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:14.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:14.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:14.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:14.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:14.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:14.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:14.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:14.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:14.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:14.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:15.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:15.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:15.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:15.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:15.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:15.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:15.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:15.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:15.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:15.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:16.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:16.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:16.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:16.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:16.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:16.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:16.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:16.828 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone2, test-zone1} 2020-06-03T19:11:16.828 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone3, test-zone1} 2020-06-03T19:11:16.837 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:16.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:16.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:16.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:16.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:16.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:16.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:16.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:16.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:16.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:16.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:16.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:16.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:16.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:16.856 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:16.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:16.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:17.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:17.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:17.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:17.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:17.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:17.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:17.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:17.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:17.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:17.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:17.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:18.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:18.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:18.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:18.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:18.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:18.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:18.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:18.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:18.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:19.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:19.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:19.309 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone3, test-zone1} 2020-06-03T19:11:19.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-59 zones={test-zone3, test-zone2} 2020-06-03T19:11:19.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:19.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:19.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:19.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:19.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:19.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:19.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:19.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:19.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:19.349 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:19.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:19.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:19.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:19.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:19.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:19.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:19.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:19.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:19.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:19.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:19.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:19.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:19.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:19.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:19.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:20.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:20.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:20.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:20.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:20.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:20.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:20.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:20.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:20.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:21.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:21.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:21.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:21.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:21.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:21.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:21.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:21.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-59 zones={test-zone3, test-zone2} 2020-06-03T19:11:21.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:21.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:21.709 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:21.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:21.753 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:21.993 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.60.1"},{"key":4,"val":"00000000006.60.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.49.1"},{"key":7,"val":"00000000006.49.5"},{"key":8,"val":"00000000004.37.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.61.1"}]} 2020-06-03T19:11:21.993 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:21.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:21.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:21.995 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:22.039 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:22.040 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:22.317 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:10:54.056621Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:48.281515Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.60.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:54.055597Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:48.283948Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.49.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1","timestamp":"2020-06-03T19:10:48.277010Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-03T19:10:54.098481Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:10:48.281478Z"}}}] 2020-06-03T19:11:22.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:22.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.319 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:22.359 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:22.359 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:22.603 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.60.1"},{"key":4,"val":"00000000006.60.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.49.1"},{"key":7,"val":"00000000006.49.5"},{"key":8,"val":"00000000004.37.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.61.1"}]} 2020-06-03T19:11:22.603 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:22.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.605 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:22.689 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:22.689 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:22.938 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:10:54.087306Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:48.269975Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.60.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:54.088010Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:48.278542Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.49.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1","timestamp":"2020-06-03T19:10:48.278504Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-03T19:10:54.087662Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:10:48.269990Z"}}}] 2020-06-03T19:11:22.939 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:22.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:22.941 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:22.980 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:22.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:23.224 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.50.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.60.1"},{"key":4,"val":"00000000006.60.5"},{"key":5,"val":"00000000004.60.1"},{"key":6,"val":"00000000004.49.1"},{"key":7,"val":"00000000006.49.5"},{"key":8,"val":"00000000004.37.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.61.1"}]} 2020-06-03T19:11:23.225 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:23.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.226 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:23.272 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:23.273 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:23.547 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:10:47.159711Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.55.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:47.160564Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.60.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1","timestamp":"2020-06-03T19:10:47.160252Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:47.161192Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.49.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1","timestamp":"2020-06-03T19:10:47.161148Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-03T19:10:47.159849Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:10:47.161727Z"}}}] 2020-06-03T19:11:23.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-60 2020-06-03T19:11:23.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.549 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:23.638 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:23.638 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:23.887 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.92.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:23.888 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:11:23.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:23.890 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:23.941 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:23.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:24.217 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:24.218 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:11:24.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.219 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:24.260 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:24.261 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:24.511 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.92.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:24.512 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:11:24.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.514 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:24.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:24.562 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:24.795 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:24.796 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-60 2020-06-03T19:11:24.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:24.798 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:24.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:25.084 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.77.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:25.085 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:11:25.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.086 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.60.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:25.431 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:11:25.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.433 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:25.739 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.77.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:25.739 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:11:25.740 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:25.741 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:25.789 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:25.789 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:26.034 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.338253Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5","timestamp":"2020-06-03T19:10:48.285711Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.60.5","timestamp":"2020-06-03T19:10:48.232927Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5","timestamp":"2020-06-03T19:10:48.398543Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:26.034 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-60 2020-06-03T19:11:26.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone2, test-zone1} 2020-06-03T19:11:26.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:26.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:26.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:26.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:26.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:26.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:26.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:26.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:26.057 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:26.058 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:26.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:26.067 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:26.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:26.068 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:26.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:26.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:26.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:26.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:26.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:26.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:26.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:26.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:26.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:26.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:26.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:26.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:26.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:26.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:26.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:27.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:27.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:27.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:27.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:27.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:27.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:27.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:27.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:27.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:27.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:28.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:28.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:28.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:28.320 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone2, test-zone1} 2020-06-03T19:11:28.321 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone3, test-zone1} 2020-06-03T19:11:28.330 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:28.344 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:28.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:28.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:28.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:28.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:28.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:28.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:28.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:28.348 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:28.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:28.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:28.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:28.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:28.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:28.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:28.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:28.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:28.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:28.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:28.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:28.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:28.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:28.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:28.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:29.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:29.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:29.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:29.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:29.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:29.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:29.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:29.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:29.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:29.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:29.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:30.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:30.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:30.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:30.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:30.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:30.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:30.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone3, test-zone1} 2020-06-03T19:11:30.642 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-60 zones={test-zone3, test-zone2} 2020-06-03T19:11:30.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:30.661 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:30.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:30.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:30.662 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:30.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:30.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:30.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:30.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:30.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:30.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:30.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:30.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:30.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:30.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:30.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:30.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:30.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:30.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:30.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:30.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:30.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:31.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:31.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:31.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:31.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:31.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:31.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:31.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:31.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:31.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:31.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:31.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:32.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:32.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:32.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:32.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:32.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:32.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:32.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:32.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:32.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:32.942 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-60 zones={test-zone3, test-zone2} 2020-06-03T19:11:32.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:32.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:32.944 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:32.985 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:32.986 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:33.229 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.53.1"},{"key":4,"val":"00000000006.69.5"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.70.5"},{"key":8,"val":"00000000004.66.1"},{"key":9,"val":"00000000004.46.1"},{"key":10,"val":"00000000004.49.1"}]} 2020-06-03T19:11:33.230 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:33.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.232 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:33.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:33.288 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:33.554 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:10:54.056641Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-03T19:10:48.284008Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.69.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-03T19:10:54.056139Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-03T19:10:48.284065Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1","timestamp":"2020-06-03T19:10:48.291962Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:54.056310Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:48.291992Z"}}}] 2020-06-03T19:11:33.554 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:33.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.556 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:33.599 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:33.599 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:33.835 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.53.1"},{"key":4,"val":"00000000006.69.5"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.70.5"},{"key":8,"val":"00000000004.66.1"},{"key":9,"val":"00000000004.46.1"},{"key":10,"val":"00000000004.49.1"}]} 2020-06-03T19:11:33.836 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:33.837 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:33.838 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:33.920 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:33.920 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:34.168 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:10:54.126393Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-03T19:10:48.305587Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.69.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-03T19:10:54.126368Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-03T19:10:48.259819Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1","timestamp":"2020-06-03T19:10:48.251694Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:54.084193Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:48.250833Z"}}}] 2020-06-03T19:11:34.169 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:34.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.171 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:34.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:34.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:34.475 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.52.1"},{"key":2,"val":"00000000006.64.5"},{"key":3,"val":"00000000004.53.1"},{"key":4,"val":"00000000006.69.5"},{"key":5,"val":"00000000004.67.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.70.5"},{"key":8,"val":"00000000004.66.1"},{"key":9,"val":"00000000004.46.1"},{"key":10,"val":"00000000004.49.1"}]} 2020-06-03T19:11:34.475 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:34.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.477 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:34.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:34.522 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:34.803 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:10:47.983383Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-03T19:10:47.983756Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.69.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-03T19:10:47.983993Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-03T19:10:47.981812Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1","timestamp":"2020-06-03T19:10:47.986201Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:10:47.984776Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-03T19:10:47.986303Z"}}}] 2020-06-03T19:11:34.819 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-61 2020-06-03T19:11:34.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:34.822 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:34.869 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:34.870 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:35.124 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.69.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:35.124 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:11:35.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.126 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:35.170 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:35.171 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:35.437 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.74.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:35.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:11:35.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.440 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:35.481 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:35.755 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000002.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.69.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:35.755 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:11:35.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:35.757 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:35.801 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:35.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:36.059 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.69.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.74.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:36.060 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-61 2020-06-03T19:11:36.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.062 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:36.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:36.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:36.393 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:36.394 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:11:36.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.395 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:36.481 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:36.482 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:36.738 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:36.738 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:11:36.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:36.740 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:36.780 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:37.014 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000002.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.77.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:37.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:11:37.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:37.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:37.017 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:37.062 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:37.063 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:37.333 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5","timestamp":"2020-06-03T19:10:48.597095Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.77.5","timestamp":"2020-06-03T19:10:48.543068Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5","timestamp":"2020-06-03T19:10:48.477612Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-03T19:10:48.649463Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:37.333 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-61 2020-06-03T19:11:37.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone2, test-zone1} 2020-06-03T19:11:37.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:37.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:37.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:37.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:37.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:37.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:37.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:37.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:37.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:37.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:37.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:37.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:37.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:37.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:37.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:37.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:37.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:37.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:37.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:37.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:37.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:37.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:38.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:38.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:38.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:38.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:38.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:38.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:38.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:38.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:38.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:38.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:39.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:39.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:39.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:39.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:39.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:39.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:39.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:39.748 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone2, test-zone1} 2020-06-03T19:11:39.749 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone3, test-zone1} 2020-06-03T19:11:39.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:39.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:39.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:39.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:39.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:39.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:39.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:39.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:39.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:39.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:39.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:39.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:39.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:39.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:39.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:39.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:39.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:39.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:39.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:39.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:39.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:40.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:40.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:40.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:40.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:40.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:40.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:40.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:40.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:41.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:41.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:41.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:41.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:41.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:41.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:41.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:41.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:41.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:41.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:42.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:42.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:42.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone3, test-zone1} 2020-06-03T19:11:42.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-61 zones={test-zone3, test-zone2} 2020-06-03T19:11:42.275 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:11:42.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:42.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:42.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:42.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:42.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:42.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:42.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:42.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:42.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:11:42.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:11:42.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:11:42.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:11:42.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:11:42.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:11:42.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:11:42.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:11:42.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:11:42.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:42.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:42.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:11:42.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:42.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:42.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:11:42.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:42.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:43.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:11:43.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:43.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:43.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:11:43.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:43.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:43.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:11:43.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:43.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:43.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:11:44.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:44.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:44.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:11:44.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:44.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:44.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:11:44.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-61 zones={test-zone3, test-zone2} 2020-06-03T19:11:44.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:44.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:44.629 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:44.668 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:44.668 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:44.901 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211468.834204_398.2","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211447.992362_579.3","last_update":"2020-06-03T19:10:47.992362Z"},{"marker":"1_1591211468.834204_540.1","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211468.834204_488.1","last_update":"2020-06-03T19:11:08.834204Z"}] 2020-06-03T19:11:44.901 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:11:44.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:44.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:44.903 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:44.947 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:44.948 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:45.197 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_398.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211447.992362_579.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:47.992362Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_540.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:45.197 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:11:45.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.199 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:45.494 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211468.834204_398.2","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211447.992362_579.3","last_update":"2020-06-03T19:10:47.992362Z"},{"marker":"1_1591211468.834204_540.1","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211468.834204_488.1","last_update":"2020-06-03T19:11:08.834204Z"}] 2020-06-03T19:11:45.495 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:11:45.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.496 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:45.578 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:45.579 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:45.821 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_398.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211447.992362_579.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:47.992362Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_540.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:45.821 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:11:45.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:45.823 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:45.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:45.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:46.148 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211468.834204_398.2","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211447.992362_579.3","last_update":"2020-06-03T19:10:47.992362Z"},{"marker":"1_1591211468.834204_540.1","last_update":"2020-06-03T19:11:08.834204Z"},{"marker":"1_1591211468.834204_488.1","last_update":"2020-06-03T19:11:08.834204Z"}] 2020-06-03T19:11:46.149 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:11:46.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.150 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:46.192 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:46.193 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:46.465 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11569358039035221345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_398.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211447.992362_579.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:47.992362Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_540.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.834204_488.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.834204Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:46.466 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:11:46.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.468 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:46.548 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:46.549 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:46.794 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591211468.671920_341.1","last_update":"2020-06-03T19:11:08.671920Z"},{"marker":"1_1591211417.719840_496.1","last_update":"2020-06-03T19:10:17.719840Z"},{"marker":"1_1591211468.587207_385.1","last_update":"2020-06-03T19:11:08.587207Z"},{"marker":"1_1591211468.632518_377.1","last_update":"2020-06-03T19:11:08.632518Z"}] 2020-06-03T19:11:46.795 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:11:46.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:46.797 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:46.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:46.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:47.090 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4441235937469265296},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.671920_341.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.671920Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.719840_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.719840Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.587207_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.587207Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.632518_377.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.632518Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:47.090 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:11:47.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.093 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:47.133 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:47.133 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:47.390 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591211468.671920_341.1","last_update":"2020-06-03T19:11:08.671920Z"},{"marker":"1_1591211417.719840_496.1","last_update":"2020-06-03T19:10:17.719840Z"},{"marker":"1_1591211468.587207_385.1","last_update":"2020-06-03T19:11:08.587207Z"},{"marker":"1_1591211468.632518_377.1","last_update":"2020-06-03T19:11:08.632518Z"}] 2020-06-03T19:11:47.391 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:11:47.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.393 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:47.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:47.436 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:47.680 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6026847009502321472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.671920_341.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.671920Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.719840_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.719840Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.587207_385.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.587207Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.632518_377.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.632518Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:47.681 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:11:47.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:47.683 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:47.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:47.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:48.003 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211468.723969_327.1","last_update":"2020-06-03T19:11:08.723969Z"},{"marker":"1_1591211417.702753_394.1","last_update":"2020-06-03T19:10:17.702753Z"},{"marker":"1_1591211468.716997_287.1","last_update":"2020-06-03T19:11:08.716997Z"},{"marker":"1_1591211468.710722_348.1","last_update":"2020-06-03T19:11:08.710722Z"}] 2020-06-03T19:11:48.004 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:11:48.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.006 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:48.084 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.723969_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.723969Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.702753_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.702753Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.716997_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.716997Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.710722_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.710722Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:48.327 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:11:48.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.329 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:48.405 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:48.406 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:48.636 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211468.723969_327.1","last_update":"2020-06-03T19:11:08.723969Z"},{"marker":"1_1591211417.702753_394.1","last_update":"2020-06-03T19:10:17.702753Z"},{"marker":"1_1591211468.716997_287.1","last_update":"2020-06-03T19:11:08.716997Z"},{"marker":"1_1591211468.710722_348.1","last_update":"2020-06-03T19:11:08.710722Z"}] 2020-06-03T19:11:48.637 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:11:48.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:48.639 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:48.681 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:48.682 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:48.925 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":18141103562818399107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211468.723969_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.723969Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211417.702753_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:10:17.702753Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211468.716997_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.716997Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211468.710722_348.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:08.710722Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:11:48.925 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:11:48.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-06-03T19:11:48.928 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-62 2020-06-03T19:11:48.971 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-63 2020-06-03T19:11:49.039 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-64 2020-06-03T19:11:49.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:49.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:49.705 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:49.759 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:49.759 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:49.950 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:11:49.983 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211509.081063_323.1","last_update":"2020-06-03T19:11:49.081063Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:11:49.984 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:11:49.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:49.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:49.987 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:50.028 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:50.028 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:50.241 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211509.074739_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:49.074739Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:11:50.241 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211509.081063_323.1 target=1_1591211509.074739_321.1 2020-06-03T19:11:55.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.244 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:55.286 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:55.287 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:55.538 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211509.081063_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:49.081063Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:11:55.538 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:11:55.539 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:11:55.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.541 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:55.576 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:55.577 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:55.823 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211509.081063_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:49.081063Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:11:55.823 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:11:55.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:11:55.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:55.825 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:55.866 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:55.866 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:56.146 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211509.081063_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:49.081063Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:11:56.147 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:11:56.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.148 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:56.191 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:56.191 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:56.440 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.73.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.95.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:56.490 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:56.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.492 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:56.538 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:56.799 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-06-03T19:11:49.310450Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:11:49.310408Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:11:49.268510Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5","timestamp":"2020-06-03T19:11:49.506259Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:56.799 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:56.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:56.801 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:56.843 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:56.843 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:57.093 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.73.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.95.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:57.094 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:57.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.096 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:57.179 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:57.179 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:57.430 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-06-03T19:11:49.265999Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:11:49.310332Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:11:49.306458Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5","timestamp":"2020-06-03T19:11:49.554319Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:57.431 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:57.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.433 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:57.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:57.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:57.742 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.73.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.95.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:57.743 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:57.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:57.745 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:57.792 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:57.793 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:58.056 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-06-03T19:11:49.201077Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:11:49.154846Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-03T19:11:49.105789Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.95.5","timestamp":"2020-06-03T19:11:49.252932Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:58.056 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:11:58.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.058 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:58.101 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:58.101 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:58.376 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.79.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.71.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.62.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:58.376 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:11:58.377 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.378 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:58.442 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:58.442 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:58.688 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-03T19:11:49.460706Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5","timestamp":"2020-06-03T19:11:49.738336Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-03T19:11:49.458919Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.651147Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:58.689 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:11:58.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.691 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:58.750 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:58.751 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:58.982 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.79.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.71.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.62.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:58.982 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:11:58.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:58.984 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:59.030 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:59.036 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:59.276 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-03T19:11:49.459380Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5","timestamp":"2020-06-03T19:11:49.734421Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-03T19:11:49.502464Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.734430Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:59.277 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:11:59.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.278 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:59.362 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:59.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:59.592 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.57.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:11:59.592 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:11:59.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.595 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:11:59.680 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:11:59.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:11:59.942 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:11:49.582338Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:11:49.538801Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:11:49.538747Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.745617Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:11:59.943 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:11:59.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:11:59.945 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:00.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:00.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:00.234 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.57.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:00.235 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:00.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:00.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:00.237 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:00.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:00.283 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:00.552 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:11:49.578361Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:11:49.538020Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:11:49.537501Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.740868Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:00.552 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:00.553 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone2, test-zone1} 2020-06-03T19:12:00.565 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:00.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:00.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:00.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:00.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:00.573 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:00.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:00.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:00.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:00.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:00.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:00.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:00.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:00.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:00.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:01.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:01.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:01.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:01.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:01.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:01.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:01.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:01.732 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone2, test-zone1} 2020-06-03T19:12:01.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone3, test-zone1} 2020-06-03T19:12:01.742 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:01.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:01.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:01.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:01.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:01.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:01.758 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:01.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:01.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:01.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:02.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:02.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:02.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:02.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:02.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:02.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:02.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:02.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:02.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:02.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone3, test-zone1} 2020-06-03T19:12:02.884 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone3, test-zone2} 2020-06-03T19:12:02.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:02.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:02.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:02.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:02.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:02.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:02.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:02.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:02.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:02.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:02.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:02.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:03.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:03.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:03.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:03.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:03.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:03.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:03.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:03.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:03.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:03.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:04.064 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone3, test-zone2} 2020-06-03T19:12:04.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.067 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:04.109 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:04.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:04.356 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:04.357 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:04.357 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.358 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:04.403 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:04.403 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:04.684 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-03T19:11:49.910372Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-03T19:11:49.910392Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:11:49.746369Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-03T19:11:49.910381Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:04.685 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:04.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.687 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:04.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:04.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:04.973 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:04.974 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:04.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:04.975 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:05.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:05.338 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-03T19:11:49.914294Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-03T19:11:49.867803Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:11:49.706274Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-03T19:11:49.914317Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:05.339 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:05.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.341 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:05.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:05.649 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:05.650 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:05.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.651 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:05.695 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:05.973 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-03T19:11:49.400859Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-03T19:11:49.353740Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-03T19:11:49.305724Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-03T19:11:49.449419Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:05.973 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:05.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:05.975 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:06.056 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:06.057 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:06.321 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.101.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:06.322 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:06.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.323 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:06.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:06.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:06.618 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:11:49.502404Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-03T19:11:49.504254Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-03T19:11:49.458941Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-03T19:11:49.694284Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:06.619 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:06.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.620 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:06.662 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:06.662 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:06.932 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.101.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:06.933 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:06.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:06.935 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:06.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:06.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:07.233 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-03T19:11:49.502498Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-03T19:11:49.457702Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-03T19:11:49.457481Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-03T19:11:49.734396Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:07.234 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:07.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.236 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:07.274 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:07.275 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:07.518 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:07.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:12:07.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.520 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:07.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:07.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:07.865 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5","timestamp":"2020-06-03T19:11:49.782290Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-03T19:11:49.744926Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:11:49.745592Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-03T19:11:50.022296Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:07.866 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:12:07.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:07.868 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:07.948 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:07.948 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:08.184 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.60.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:08.185 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:12:08.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:08.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:08.187 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:08.237 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:08.238 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:08.499 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5","timestamp":"2020-06-03T19:11:49.780459Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-03T19:11:49.743045Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:11:49.826216Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-03T19:11:50.026286Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:08.500 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:12:08.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone2, test-zone1} 2020-06-03T19:12:08.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:08.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:08.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:08.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:08.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:08.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:08.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:08.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:08.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:08.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:08.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:08.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:08.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:08.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:08.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:09.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:09.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:09.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:09.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:09.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:09.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:09.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:09.793 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone2, test-zone1} 2020-06-03T19:12:09.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone3, test-zone1} 2020-06-03T19:12:09.802 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:09.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:09.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:09.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:09.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:09.810 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:09.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:09.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:09.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:09.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:09.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:09.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:10.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:10.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:10.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:10.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:10.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:10.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:10.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:10.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:10.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:10.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:11.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone3, test-zone1} 2020-06-03T19:12:11.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone3, test-zone2} 2020-06-03T19:12:11.030 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:11.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:11.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:11.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:11.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:11.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:11.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:11.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:11.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:11.049 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:11.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:11.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:11.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:11.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:11.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:11.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:11.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:11.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:11.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:11.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:11.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:12.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:12.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone3, test-zone2} 2020-06-03T19:12:12.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.207 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:12.251 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:12.252 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:12.502 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.77.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:12.502 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:12.503 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.504 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:12.550 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:12.789 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:11:50.146313Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-03T19:11:50.150545Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-06-03T19:11:49.910340Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-03T19:11:50.150579Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:12.790 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:12.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:12.791 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:12.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:12.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:13.072 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.77.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:13.072 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:13.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.074 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:13.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:13.420 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:11:50.150355Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-03T19:11:50.154605Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-06-03T19:11:49.906272Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-03T19:11:50.102319Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:13.420 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:13.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.422 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:13.461 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:13.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:13.689 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.77.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:13.689 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:13.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:13.691 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:13.736 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:14.006 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:11:49.600951Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-03T19:11:49.546256Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-06-03T19:11:49.498295Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-03T19:11:49.650703Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:14.007 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:12:14.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.009 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:14.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:14.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:14.347 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.103.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.72.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:14.348 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:12:14.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.350 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:14.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:14.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:14.643 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:11:50.094486Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5","timestamp":"2020-06-03T19:11:50.094420Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5","timestamp":"2020-06-03T19:11:49.890323Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:11:50.094459Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:14.644 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:12:14.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.646 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:14.689 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:14.689 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:14.923 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000001.63.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.103.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.72.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:14.924 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:12:14.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:14.925 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:14.972 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:14.973 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:15.216 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:11:50.090325Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5","timestamp":"2020-06-03T19:11:50.094409Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5","timestamp":"2020-06-03T19:11:49.894359Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:11:50.094389Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:15.217 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:12:15.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.219 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:15.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:15.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:15.554 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.65.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.83.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:15.555 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:12:15.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.558 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:15.639 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:15.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:15.900 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-03T19:11:49.786368Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-06-03T19:11:49.786400Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.578282Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.746479Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:15.901 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:12:15.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:15.904 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:15.990 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:15.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:16.225 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000001.65.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.83.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:12:16.225 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:12:16.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:16.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:16.227 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:16.272 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:16.273 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:16.546 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-03T19:11:49.822241Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-06-03T19:11:49.782279Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.578317Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-03T19:11:49.782315Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:12:16.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:12:16.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone2, test-zone1} 2020-06-03T19:12:16.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:16.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:16.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:16.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:16.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:16.574 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:16.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:16.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:16.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:16.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:16.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:16.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:16.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:16.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:16.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:17.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:17.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:17.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:17.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:17.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:17.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:17.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:17.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone2, test-zone1} 2020-06-03T19:12:17.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone3, test-zone1} 2020-06-03T19:12:17.762 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:17.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:17.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:17.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:17.770 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:17.772 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:17.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:17.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:17.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:17.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:17.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:17.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:17.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:18.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:18.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:18.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:18.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:18.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:18.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:18.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:18.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:18.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:18.928 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone3, test-zone1} 2020-06-03T19:12:18.929 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone3, test-zone2} 2020-06-03T19:12:18.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:18.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:18.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:18.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:18.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:18.950 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:18.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:18.957 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:18.958 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:18.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:18.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:19.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:19.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:19.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:19.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:19.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:19.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:19.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:19.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:19.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:19.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:20.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:20.131 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone3, test-zone2} 2020-06-03T19:12:20.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.133 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:20.176 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:20.177 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:20.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.510 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:20.589 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:20.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:20.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'rhmmtn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'rhmmtn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:20.913 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket rhmmtn-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:20.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:20.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:21.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:21.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:21.323 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:21.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:21.403 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:21.606 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:12:21.638 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211541.269066_332.1","last_update":"2020-06-03T19:12:21.269066Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:12:21.638 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:12:21.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:21.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:21.640 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:21.680 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:21.680 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:21.908 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211509.081063_323.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:11:49.081063Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:12:21.908 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1591211541.269066_332.1 target=1_1591211509.081063_323.1 2020-06-03T19:12:26.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:26.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:26.911 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:26.955 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:26.955 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:27.199 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211541.269066_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:21.269066Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:12:27.200 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:12:27.200 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:12:27.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:27.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:27.202 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:27.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:27.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:27.481 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211541.269066_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:21.269066Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:12:27.482 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:12:27.483 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:12:27.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:27.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:27.486 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:27.527 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:27.528 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:27.806 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211541.269066_332.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:21.269066Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:12:27.806 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:12:28.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:28.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:28.424 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:28.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:28.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:28.769 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:28.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:28.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:29.162 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'rhmmtn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:29.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'rhmmtn-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:12:29.164 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket rhmmtn-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:12:29.245 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:29.246 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:29.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:29.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:29.540 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:29.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:29.621 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:29.848 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.79.1"},{"key":1,"val":"00000000004.72.1"},{"key":2,"val":"00000000006.86.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.61.1"},{"key":6,"val":"00000000004.73.1"},{"key":7,"val":"00000000006.103.1"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.50.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-06-03T19:12:29.848 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:29.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:29.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:29.850 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:29.901 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:29.901 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:30.128 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:29.282443Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:29.281140Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:29.514388Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:29.330219Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:29.322337Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:29.471754Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:29.283431Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:29.283449Z"}}}] 2020-06-03T19:12:30.129 WARNING:rgw_multi.tests:bucket rhmmtn-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.72.1 target=, shard=5 master=00000000004.61.1 target=, shard=9 master=00000000004.50.1 target= 2020-06-03T19:12:35.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.132 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:35.455 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:29.282443Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-03T19:12:34.090349Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:29.281140Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:29.514388Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:29.330219Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:12:34.050106Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:29.322337Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:29.471754Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:29.283431Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:34.049137Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:29.283449Z"}}}] 2020-06-03T19:12:35.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:35.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.458 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:35.749 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.79.1"},{"key":1,"val":"00000000004.72.1"},{"key":2,"val":"00000000006.86.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.61.1"},{"key":6,"val":"00000000004.73.1"},{"key":7,"val":"00000000006.103.1"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.50.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-06-03T19:12:35.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:35.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:35.751 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:35.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:35.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:36.082 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:34.082612Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-03T19:12:34.130406Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:34.162334Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:34.037169Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:34.091308Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:12:34.130447Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:34.166375Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:34.078402Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:34.035968Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:34.130438Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:34.166349Z"}}}] 2020-06-03T19:12:36.083 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:36.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:36.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:36.085 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:36.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:36.166 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:36.416 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.79.1"},{"key":1,"val":"00000000004.72.1"},{"key":2,"val":"00000000006.86.1"},{"key":3,"val":"00000000004.80.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.61.1"},{"key":6,"val":"00000000004.73.1"},{"key":7,"val":"00000000006.103.1"},{"key":8,"val":"00000000004.54.1"},{"key":9,"val":"00000000004.50.1"},{"key":10,"val":"00000000004.62.1"}]} 2020-06-03T19:12:36.416 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:36.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:36.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:36.418 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:36.466 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:36.466 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:36.737 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:28.722309Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:28.722160Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:28.723762Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:28.724405Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:28.724541Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:28.722697Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:28.723229Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:28.724655Z"}}}] 2020-06-03T19:12:36.738 WARNING:rgw_multi.tests:bucket rhmmtn-62 zone test-zone4 behind zone test-zone1: shard=1 master=00000000004.72.1 target=, shard=5 master=00000000004.61.1 target=, shard=9 master=00000000004.50.1 target= 2020-06-03T19:12:41.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:41.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:41.741 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:41.788 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:41.789 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:42.061 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:28.722309Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:28.722160Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:28.723762Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:28.724405Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:28.724541Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:28.722697Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:28.723229Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:28.724655Z"}}}] 2020-06-03T19:12:42.061 WARNING:rgw_multi.tests:bucket rhmmtn-62 zone test-zone4 behind zone test-zone1: shard=1 master=00000000004.72.1 target=, shard=5 master=00000000004.61.1 target=, shard=9 master=00000000004.50.1 target= 2020-06-03T19:12:47.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.065 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:47.110 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:47.376 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.79.1","timestamp":"2020-06-03T19:12:28.722309Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-03T19:12:28.722487Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.86.1","timestamp":"2020-06-03T19:12:28.722160Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-03T19:12:28.723762Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.1","timestamp":"2020-06-03T19:12:28.724405Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1","timestamp":"2020-06-03T19:12:28.722797Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:28.724541Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.103.1","timestamp":"2020-06-03T19:12:28.722697Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:28.723229Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:28.723887Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1","timestamp":"2020-06-03T19:12:28.724655Z"}}}] 2020-06-03T19:12:47.377 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-62 2020-06-03T19:12:47.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.379 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:47.704 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.85.5"},{"key":1,"val":"00000000002.79.1"},{"key":2,"val":"00000000004.81.5"},{"key":3,"val":"00000000002.103.1"},{"key":4,"val":"00000000004.80.5"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.81.1"},{"key":7,"val":"00000000004.93.5"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.47.1"},{"key":10,"val":"00000000002.77.1"}]} 2020-06-03T19:12:47.705 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:12:47.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:47.707 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:47.756 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:48.020 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.85.5","timestamp":"2020-06-03T19:12:42.340666Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1","timestamp":"2020-06-03T19:12:42.345559Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5","timestamp":"2020-06-03T19:12:42.387148Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.103.1","timestamp":"2020-06-03T19:12:42.378368Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.5","timestamp":"2020-06-03T19:12:42.354873Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.387087Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1","timestamp":"2020-06-03T19:12:42.340749Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.5","timestamp":"2020-06-03T19:12:42.345483Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-03T19:12:42.422347Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-03T19:12:42.345493Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-03T19:12:42.387063Z"}}}] 2020-06-03T19:12:48.021 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:12:48.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.023 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:48.065 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:48.066 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:48.320 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.85.5"},{"key":1,"val":"00000000002.79.1"},{"key":2,"val":"00000000004.81.5"},{"key":3,"val":"00000000002.103.1"},{"key":4,"val":"00000000004.80.5"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.81.1"},{"key":7,"val":"00000000004.93.5"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.47.1"},{"key":10,"val":"00000000002.77.1"}]} 2020-06-03T19:12:48.320 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:12:48.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.322 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:48.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:48.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:48.610 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.85.5","timestamp":"2020-06-03T19:12:42.325515Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1","timestamp":"2020-06-03T19:12:42.342789Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.81.5","timestamp":"2020-06-03T19:12:42.345667Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.103.1","timestamp":"2020-06-03T19:12:42.345652Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.5","timestamp":"2020-06-03T19:12:42.374287Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.386344Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1","timestamp":"2020-06-03T19:12:42.363433Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.5","timestamp":"2020-06-03T19:12:42.363383Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-03T19:12:42.471313Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-03T19:12:42.363312Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-03T19:12:42.406772Z"}}}] 2020-06-03T19:12:48.611 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-62 2020-06-03T19:12:48.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.612 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:48.653 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:48.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:48.895 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.69.5"},{"key":1,"val":"00000000002.68.1"},{"key":2,"val":"00000000004.78.5"},{"key":3,"val":"00000000002.71.1"},{"key":4,"val":"00000000004.73.5"},{"key":5,"val":"00000000002.48.1"},{"key":6,"val":"00000000002.70.1"},{"key":7,"val":"00000000004.93.5"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.44.1"},{"key":10,"val":"00000000002.51.1"}]} 2020-06-03T19:12:48.895 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:48.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:48.897 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:48.979 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:48.980 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:49.259 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:12:42.222495Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1","timestamp":"2020-06-03T19:12:42.196335Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.78.5","timestamp":"2020-06-03T19:12:42.213339Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-03T19:12:42.242592Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.5","timestamp":"2020-06-03T19:12:42.177075Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1","timestamp":"2020-06-03T19:12:42.195747Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-06-03T19:12:42.205536Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.5","timestamp":"2020-06-03T19:12:42.254733Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-03T19:12:42.226345Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-03T19:12:42.254407Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-03T19:12:42.250374Z"}}}] 2020-06-03T19:12:49.260 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:49.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:49.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:49.263 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:49.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:49.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:49.568 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.69.5"},{"key":1,"val":"00000000002.68.1"},{"key":2,"val":"00000000004.78.5"},{"key":3,"val":"00000000002.71.1"},{"key":4,"val":"00000000004.73.5"},{"key":5,"val":"00000000002.48.1"},{"key":6,"val":"00000000002.70.1"},{"key":7,"val":"00000000004.93.5"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.44.1"},{"key":10,"val":"00000000002.51.1"}]} 2020-06-03T19:12:49.569 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:49.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:49.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:49.571 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:49.869 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:12:42.197829Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1","timestamp":"2020-06-03T19:12:42.198203Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.78.5","timestamp":"2020-06-03T19:12:42.211477Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-03T19:12:42.198268Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.5","timestamp":"2020-06-03T19:12:42.197930Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1","timestamp":"2020-06-03T19:12:42.203422Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-06-03T19:12:42.198247Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.5","timestamp":"2020-06-03T19:12:42.211624Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-03T19:12:42.215097Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-03T19:12:42.211816Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-03T19:12:42.202971Z"}}}] 2020-06-03T19:12:49.869 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-62 2020-06-03T19:12:49.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone2, test-zone1} 2020-06-03T19:12:49.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:49.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:49.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:49.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:49.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:49.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:49.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:49.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:49.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:49.895 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:49.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:49.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:49.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:49.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:49.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:49.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:49.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:49.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:49.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:49.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:50.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:50.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:50.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:50.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:50.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:50.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:50.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:50.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:50.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:51.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:51.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:51.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:51.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:51.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:51.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:51.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:51.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:51.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:51.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:51.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:52.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:52.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:52.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone2, test-zone1} 2020-06-03T19:12:52.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone3, test-zone1} 2020-06-03T19:12:52.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:52.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:52.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:52.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:52.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:52.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:52.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:52.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:52.280 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:52.281 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:52.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:52.291 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:52.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:52.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:52.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:52.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:52.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:52.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:52.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:52.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:52.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:52.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:52.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:52.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:52.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:52.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:53.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:53.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:53.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:53.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:53.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:53.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:53.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:53.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:53.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:54.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:54.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:54.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:54.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:54.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:54.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:54.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:54.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone3, test-zone1} 2020-06-03T19:12:54.704 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-62 zones={test-zone3, test-zone2} 2020-06-03T19:12:54.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:12:54.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:54.723 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:54.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:54.724 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:54.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:54.725 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:54.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:54.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:54.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:12:54.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:12:54.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:12:54.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:12:54.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:12:54.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:12:54.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:12:54.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:12:54.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:12:54.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:54.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:54.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:12:55.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:55.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:55.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:12:55.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:55.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:55.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:12:55.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:55.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:55.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:12:55.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:55.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:56.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:12:56.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:56.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:56.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:12:56.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:56.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:56.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:12:56.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:57.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:57.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:12:57.299 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-62 zones={test-zone3, test-zone2} 2020-06-03T19:12:57.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.306 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:57.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:57.607 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.29.1"},{"key":2,"val":"00000000006.79.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.95.5"},{"key":5,"val":"00000000004.70.1"},{"key":6,"val":"00000000004.50.1"},{"key":7,"val":"00000000006.76.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.105.1"},{"key":10,"val":"00000000004.87.1"}]} 2020-06-03T19:12:57.607 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:57.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.609 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:57.655 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:57.656 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:57.912 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-03T19:12:28.124150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1","timestamp":"2020-06-03T19:12:34.090323Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.79.5","timestamp":"2020-06-03T19:12:28.078908Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:12:29.514424Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.95.5","timestamp":"2020-06-03T19:12:28.025484Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1","timestamp":"2020-06-03T19:12:34.049751Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:29.282730Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.76.5","timestamp":"2020-06-03T19:12:28.171984Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:12:29.282766Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.105.1","timestamp":"2020-06-03T19:12:34.043391Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-06-03T19:12:29.281523Z"}}}] 2020-06-03T19:12:57.913 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:57.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:57.915 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:57.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:57.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:58.205 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.29.1"},{"key":2,"val":"00000000006.79.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.95.5"},{"key":5,"val":"00000000004.70.1"},{"key":6,"val":"00000000004.50.1"},{"key":7,"val":"00000000006.76.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.105.1"},{"key":10,"val":"00000000004.87.1"}]} 2020-06-03T19:12:58.206 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:58.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.207 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:58.290 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:58.291 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:58.549 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-03T19:12:28.124150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1","timestamp":"2020-06-03T19:12:34.130465Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.79.5","timestamp":"2020-06-03T19:12:28.078908Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:12:34.050453Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.95.5","timestamp":"2020-06-03T19:12:28.025484Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1","timestamp":"2020-06-03T19:12:34.130455Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:34.121916Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.76.5","timestamp":"2020-06-03T19:12:28.171984Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:12:34.050427Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.105.1","timestamp":"2020-06-03T19:12:34.088323Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-06-03T19:12:34.162357Z"}}}] 2020-06-03T19:12:58.550 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:58.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.552 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:58.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:58.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:58.870 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.29.1"},{"key":2,"val":"00000000006.79.5"},{"key":3,"val":"00000000004.46.1"},{"key":4,"val":"00000000006.95.5"},{"key":5,"val":"00000000004.70.1"},{"key":6,"val":"00000000004.50.1"},{"key":7,"val":"00000000006.76.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.105.1"},{"key":10,"val":"00000000004.87.1"}]} 2020-06-03T19:12:58.871 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:58.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:58.874 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:58.920 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:58.921 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:59.183 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-03T19:12:28.124150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1","timestamp":"2020-06-03T19:12:29.111603Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.79.5","timestamp":"2020-06-03T19:12:28.078908Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.46.1","timestamp":"2020-06-03T19:12:29.112227Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.95.5","timestamp":"2020-06-03T19:12:28.025484Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1","timestamp":"2020-06-03T19:12:29.112344Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:29.112907Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.76.5","timestamp":"2020-06-03T19:12:28.171984Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-03T19:12:29.116079Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.105.1","timestamp":"2020-06-03T19:12:29.111563Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.87.1","timestamp":"2020-06-03T19:12:29.113541Z"}}}] 2020-06-03T19:12:59.184 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-63 2020-06-03T19:12:59.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.187 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:59.267 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:59.268 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:59.517 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.73.1"},{"key":1,"val":"00000000002.39.1"},{"key":2,"val":"00000000004.78.1"},{"key":3,"val":"00000000002.52.1"},{"key":4,"val":"00000000004.112.1"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.41.1"},{"key":7,"val":"00000000004.67.1"},{"key":8,"val":"00000000002.44.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.113.1"}]} 2020-06-03T19:12:59.518 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:59.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.520 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:59.575 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:59.575 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:12:59.841 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:42.402434Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1","timestamp":"2020-06-03T19:12:42.352374Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.78.1","timestamp":"2020-06-03T19:12:42.354898Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1","timestamp":"2020-06-03T19:12:42.341976Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.112.1","timestamp":"2020-06-03T19:12:42.353577Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.354606Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-03T19:12:42.353324Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-03T19:12:42.340103Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-03T19:12:42.354578Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-06-03T19:12:42.402469Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.113.1","timestamp":"2020-06-03T19:12:42.352333Z"}}}] 2020-06-03T19:12:59.842 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:12:59.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:12:59.844 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:12:59.887 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:12:59.887 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:00.123 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.73.1"},{"key":1,"val":"00000000002.39.1"},{"key":2,"val":"00000000004.78.1"},{"key":3,"val":"00000000002.52.1"},{"key":4,"val":"00000000004.112.1"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.41.1"},{"key":7,"val":"00000000004.67.1"},{"key":8,"val":"00000000002.44.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.113.1"}]} 2020-06-03T19:13:00.123 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:13:00.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.125 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:00.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:00.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:00.435 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-03T19:12:42.406727Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1","timestamp":"2020-06-03T19:12:42.373960Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.78.1","timestamp":"2020-06-03T19:12:42.429852Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1","timestamp":"2020-06-03T19:12:42.471940Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.112.1","timestamp":"2020-06-03T19:12:42.386327Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.410428Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-03T19:12:42.429815Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-03T19:12:42.414784Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-03T19:12:42.334295Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-06-03T19:12:42.368626Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.113.1","timestamp":"2020-06-03T19:12:42.370152Z"}}}] 2020-06-03T19:13:00.436 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-63 2020-06-03T19:13:00.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.438 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:00.478 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:00.478 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:00.719 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.75.5"},{"key":1,"val":"00000000002.34.1"},{"key":2,"val":"00000000004.76.5"},{"key":3,"val":"00000000002.40.1"},{"key":4,"val":"00000000004.101.5"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.50.1"},{"key":7,"val":"00000000004.66.5"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.95.1"}]} 2020-06-03T19:13:00.720 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:13:00.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:00.721 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:00.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:00.806 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:01.064 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.63.1","timestamp":"2020-06-03T19:12:42.222515Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1","timestamp":"2020-06-03T19:12:42.204353Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.63.1","timestamp":"2020-06-03T19:12:42.242560Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1","timestamp":"2020-06-03T19:12:42.204586Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.94.1","timestamp":"2020-06-03T19:12:42.180581Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1","timestamp":"2020-06-03T19:12:42.204372Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1","timestamp":"2020-06-03T19:12:42.246287Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.62.1","timestamp":"2020-06-03T19:12:42.250403Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-03T19:12:42.222444Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-06-03T19:12:42.254364Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-03T19:12:42.254714Z"}}}] 2020-06-03T19:13:01.064 WARNING:rgw_multi.tests:bucket rhmmtn-63 zone test-zone1 behind zone test-zone3: shard=0 master=00000000004.75.5 target=00000000003.63.1, shard=2 master=00000000004.76.5 target=00000000003.63.1, shard=4 master=00000000004.101.5 target=00000000003.94.1, shard=7 master=00000000004.66.5 target=00000000003.62.1 2020-06-03T19:13:06.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.067 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:06.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:06.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:06.386 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.75.5","timestamp":"2020-06-03T19:12:28.124150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1","timestamp":"2020-06-03T19:12:42.204353Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5","timestamp":"2020-06-03T19:12:28.078908Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1","timestamp":"2020-06-03T19:12:42.204586Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.101.5","timestamp":"2020-06-03T19:12:28.025484Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1","timestamp":"2020-06-03T19:12:42.204372Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1","timestamp":"2020-06-03T19:12:42.246287Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-03T19:12:28.171984Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-03T19:12:42.222444Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-06-03T19:12:42.254364Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-03T19:12:42.254714Z"}}}] 2020-06-03T19:13:06.386 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:13:06.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.388 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:06.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:06.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:06.692 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.75.5"},{"key":1,"val":"00000000002.34.1"},{"key":2,"val":"00000000004.76.5"},{"key":3,"val":"00000000002.40.1"},{"key":4,"val":"00000000004.101.5"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.50.1"},{"key":7,"val":"00000000004.66.5"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.86.1"},{"key":10,"val":"00000000002.95.1"}]} 2020-06-03T19:13:06.693 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:13:06.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:06.696 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:06.740 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:06.741 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:07.006 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.75.5","timestamp":"2020-06-03T19:12:28.124150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1","timestamp":"2020-06-03T19:12:42.198116Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.76.5","timestamp":"2020-06-03T19:12:28.078908Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1","timestamp":"2020-06-03T19:12:42.192642Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.101.5","timestamp":"2020-06-03T19:12:28.025484Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1","timestamp":"2020-06-03T19:12:42.198182Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1","timestamp":"2020-06-03T19:12:42.202911Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-03T19:12:28.171984Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-03T19:12:42.255127Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.1","timestamp":"2020-06-03T19:12:42.211603Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-03T19:12:42.255171Z"}}}] 2020-06-03T19:13:07.007 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-63 2020-06-03T19:13:07.008 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone2, test-zone1} 2020-06-03T19:13:07.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:07.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:07.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:07.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:07.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:07.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:07.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:07.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:07.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:07.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:07.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:07.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:07.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:07.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:07.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:07.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:07.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:07.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:07.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:07.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:07.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:07.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:07.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:07.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:07.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:07.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:07.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:07.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:07.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:08.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:08.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:08.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:08.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:08.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:08.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:08.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:08.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:08.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:08.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:09.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:09.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:09.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:09.332 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone2, test-zone1} 2020-06-03T19:13:09.332 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone3, test-zone1} 2020-06-03T19:13:09.342 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:09.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:09.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:09.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:09.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:09.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:09.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:09.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:09.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:09.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:09.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:09.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:09.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:09.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:09.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:09.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:09.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:09.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:09.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:09.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:09.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:09.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:09.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:09.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:09.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:10.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:10.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:10.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:10.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:10.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:10.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:10.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:10.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:10.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:10.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:11.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:11.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:11.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:11.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:11.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:11.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:11.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:11.691 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone3, test-zone1} 2020-06-03T19:13:11.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-63 zones={test-zone3, test-zone2} 2020-06-03T19:13:11.703 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:11.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:11.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:11.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:11.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:11.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:11.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:11.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:11.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:11.717 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:11.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:11.726 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:11.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:11.727 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:11.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:11.728 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:11.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:11.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:11.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:11.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:11.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:12.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:12.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:12.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:12.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:12.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:12.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:12.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:12.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:12.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:12.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:12.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:13.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:13.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:13.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:13.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:13.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:13.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:13.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:13.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:13.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:13.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:14.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-63 zones={test-zone3, test-zone2} 2020-06-03T19:13:14.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.086 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:14.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:14.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:14.367 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.67.5"},{"key":1,"val":"00000000004.77.1"},{"key":2,"val":"00000000006.56.5"},{"key":3,"val":"00000000004.187.1"},{"key":4,"val":"00000000007.93.5"},{"key":5,"val":"00000000004.50.1"},{"key":6,"val":"00000000004.54.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.89.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.90.1"}]} 2020-06-03T19:13:14.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:14.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.369 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:14.704 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.67.5","timestamp":"2020-06-03T19:12:28.331591Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-06-03T19:12:34.090357Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.56.5","timestamp":"2020-06-03T19:12:28.278843Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.187.1","timestamp":"2020-06-03T19:12:29.491678Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.93.5","timestamp":"2020-06-03T19:12:28.231689Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:34.049406Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:29.958308Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5","timestamp":"2020-06-03T19:12:28.379985Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.89.1","timestamp":"2020-06-03T19:12:29.716971Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-03T19:12:34.049677Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.1","timestamp":"2020-06-03T19:12:29.916915Z"}}}] 2020-06-03T19:13:14.705 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:14.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.707 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:14.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:14.748 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:14.991 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.67.5"},{"key":1,"val":"00000000004.77.1"},{"key":2,"val":"00000000006.56.5"},{"key":3,"val":"00000000004.187.1"},{"key":4,"val":"00000000007.93.5"},{"key":5,"val":"00000000004.50.1"},{"key":6,"val":"00000000004.54.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.89.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.90.1"}]} 2020-06-03T19:13:14.992 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:14.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:14.993 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:15.077 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:15.078 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:15.334 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.67.5","timestamp":"2020-06-03T19:12:28.331591Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-06-03T19:12:34.130474Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.56.5","timestamp":"2020-06-03T19:12:28.278843Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.187.1","timestamp":"2020-06-03T19:12:34.046448Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.93.5","timestamp":"2020-06-03T19:12:28.231689Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:34.088304Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:34.162350Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5","timestamp":"2020-06-03T19:12:28.379985Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.89.1","timestamp":"2020-06-03T19:12:34.046420Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-03T19:12:34.088272Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.1","timestamp":"2020-06-03T19:12:34.121266Z"}}}] 2020-06-03T19:13:15.334 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:15.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.336 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:15.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:15.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:15.606 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000006.67.5"},{"key":1,"val":"00000000004.77.1"},{"key":2,"val":"00000000006.56.5"},{"key":3,"val":"00000000004.187.1"},{"key":4,"val":"00000000007.93.5"},{"key":5,"val":"00000000004.50.1"},{"key":6,"val":"00000000004.54.1"},{"key":7,"val":"00000000006.73.5"},{"key":8,"val":"00000000004.89.1"},{"key":9,"val":"00000000004.65.1"},{"key":10,"val":"00000000004.90.1"}]} 2020-06-03T19:13:15.606 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:15.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.608 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:15.657 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:15.657 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:15.921 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.67.5","timestamp":"2020-06-03T19:12:28.331591Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-06-03T19:12:29.492090Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.56.5","timestamp":"2020-06-03T19:12:28.278843Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.187.1","timestamp":"2020-06-03T19:12:29.491678Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.93.5","timestamp":"2020-06-03T19:12:28.231689Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1","timestamp":"2020-06-03T19:12:29.492702Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-03T19:12:29.495107Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.5","timestamp":"2020-06-03T19:12:28.379985Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.89.1","timestamp":"2020-06-03T19:12:29.492364Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-03T19:12:29.495404Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.1","timestamp":"2020-06-03T19:12:29.493680Z"}}}] 2020-06-03T19:13:15.922 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-64 2020-06-03T19:13:15.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:15.924 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:15.964 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:15.964 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:16.222 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.73.5"},{"key":1,"val":"00000000002.82.1"},{"key":2,"val":"00000000004.51.5"},{"key":3,"val":"00000000002.185.1"},{"key":4,"val":"00000000004.119.5"},{"key":5,"val":"00000000002.43.1"},{"key":6,"val":"00000000002.47.1"},{"key":7,"val":"00000000004.78.5"},{"key":8,"val":"00000000002.82.1"},{"key":9,"val":"00000000002.69.1"},{"key":10,"val":"00000000002.83.1"}]} 2020-06-03T19:13:16.222 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:13:16.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.224 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:16.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:16.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:16.544 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.5","timestamp":"2020-06-03T19:12:28.331591Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1","timestamp":"2020-06-03T19:12:42.387130Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.5","timestamp":"2020-06-03T19:12:42.393810Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.185.1","timestamp":"2020-06-03T19:12:42.340129Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.119.5","timestamp":"2020-06-03T19:12:28.231689Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1","timestamp":"2020-06-03T19:12:42.387108Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-03T19:12:42.393842Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.78.5","timestamp":"2020-06-03T19:12:28.379985Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1","timestamp":"2020-06-03T19:12:42.394447Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1","timestamp":"2020-06-03T19:12:42.345511Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1","timestamp":"2020-06-03T19:12:42.340643Z"}}}] 2020-06-03T19:13:16.544 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:13:16.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.546 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:16.591 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:16.592 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:16.844 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":"00000000004.73.5"},{"key":1,"val":"00000000002.82.1"},{"key":2,"val":"00000000004.51.5"},{"key":3,"val":"00000000002.185.1"},{"key":4,"val":"00000000004.119.5"},{"key":5,"val":"00000000002.43.1"},{"key":6,"val":"00000000002.47.1"},{"key":7,"val":"00000000004.78.5"},{"key":8,"val":"00000000002.82.1"},{"key":9,"val":"00000000002.69.1"},{"key":10,"val":"00000000002.83.1"}]} 2020-06-03T19:13:16.844 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:13:16.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:16.846 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:17.147 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.73.5","timestamp":"2020-06-03T19:12:42.474742Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1","timestamp":"2020-06-03T19:12:42.471396Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.5","timestamp":"2020-06-03T19:12:42.471459Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.185.1","timestamp":"2020-06-03T19:12:42.414768Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.119.5","timestamp":"2020-06-03T19:12:42.471427Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1","timestamp":"2020-06-03T19:12:42.471412Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-03T19:12:42.471959Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.78.5","timestamp":"2020-06-03T19:12:42.471915Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1","timestamp":"2020-06-03T19:12:42.471443Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1","timestamp":"2020-06-03T19:12:42.414778Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1","timestamp":"2020-06-03T19:12:42.370162Z"}}}] 2020-06-03T19:13:17.148 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-64 2020-06-03T19:13:17.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.149 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:17.471 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.72.1"},{"key":1,"val":"00000000002.71.1"},{"key":2,"val":"00000000004.51.1"},{"key":3,"val":"00000000002.196.1"},{"key":4,"val":"00000000004.96.1"},{"key":5,"val":"00000000002.49.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.88.1"},{"key":8,"val":"00000000002.72.1"},{"key":9,"val":"00000000002.73.1"},{"key":10,"val":"00000000002.74.1"}]} 2020-06-03T19:13:17.472 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:13:17.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.473 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:17.787 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-03T19:12:42.180597Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-03T19:12:42.181654Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-03T19:12:42.222552Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.196.1","timestamp":"2020-06-03T19:12:42.193623Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.96.1","timestamp":"2020-06-03T19:12:42.222535Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-06-03T19:12:42.195736Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-06-03T19:12:42.181632Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.1","timestamp":"2020-06-03T19:12:42.213203Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-06-03T19:12:42.218341Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1","timestamp":"2020-06-03T19:12:42.230694Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.213296Z"}}}] 2020-06-03T19:13:17.788 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:13:17.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:17.789 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:17.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:17.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:18.121 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":"00000000004.72.1"},{"key":1,"val":"00000000002.71.1"},{"key":2,"val":"00000000004.51.1"},{"key":3,"val":"00000000002.196.1"},{"key":4,"val":"00000000004.96.1"},{"key":5,"val":"00000000002.49.1"},{"key":6,"val":"00000000002.53.1"},{"key":7,"val":"00000000004.88.1"},{"key":8,"val":"00000000002.72.1"},{"key":9,"val":"00000000002.73.1"},{"key":10,"val":"00000000002.74.1"}]} 2020-06-03T19:13:18.121 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:13:18.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:18.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:18.123 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:18.449 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-03T19:12:42.198150Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-03T19:12:42.230230Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-03T19:12:42.193380Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.196.1","timestamp":"2020-06-03T19:12:42.195694Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.96.1","timestamp":"2020-06-03T19:12:42.211454Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-06-03T19:12:42.202985Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.53.1","timestamp":"2020-06-03T19:12:42.203019Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.1","timestamp":"2020-06-03T19:12:42.211498Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-06-03T19:12:42.211390Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1","timestamp":"2020-06-03T19:12:42.198235Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-03T19:12:42.211433Z"}}}] 2020-06-03T19:13:18.450 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-64 2020-06-03T19:13:18.451 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone2, test-zone1} 2020-06-03T19:13:18.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:18.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:18.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:18.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:18.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:18.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:18.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:18.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:18.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:18.476 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:18.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:18.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:18.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:18.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:18.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:18.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:18.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:18.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:18.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:18.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:18.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:18.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:18.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:18.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:19.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:19.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:19.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:19.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:19.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:19.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:19.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:19.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:19.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:19.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:19.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:20.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:20.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:20.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:20.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:20.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:20.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:20.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:20.768 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone2, test-zone1} 2020-06-03T19:13:20.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone3, test-zone1} 2020-06-03T19:13:20.779 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:20.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:20.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:20.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:20.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:20.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:20.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:20.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:20.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:20.790 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:20.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:20.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:20.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:20.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:20.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:20.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:20.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:20.800 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:20.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:20.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:20.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:21.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:21.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:21.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:21.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:21.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:21.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:21.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:21.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:21.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:21.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:22.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:22.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:22.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:22.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:22.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:22.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:22.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:22.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:22.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:22.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:23.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:23.121 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone3, test-zone1} 2020-06-03T19:13:23.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-64 zones={test-zone3, test-zone2} 2020-06-03T19:13:23.132 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:23.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:23.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:23.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:23.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:23.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:23.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:23.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:23.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:23.144 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:23.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-03T19:13:23.152 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-03T19:13:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-03T19:13:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-03T19:13:23.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-03T19:13:23.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-03T19:13:23.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-03T19:13:23.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-03T19:13:23.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:23.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:23.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-03T19:13:23.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:23.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:23.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-03T19:13:23.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:23.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:23.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-03T19:13:24.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:24.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:24.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-03T19:13:24.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:24.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:24.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-03T19:13:24.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:24.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:24.731 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-03T19:13:24.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:24.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:25.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-03T19:13:25.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:25.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:25.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-03T19:13:25.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-64 zones={test-zone3, test-zone2} 2020-06-03T19:13:25.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.407 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:25.674 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211578.847144_456.1","last_update":"2020-06-03T19:12:58.847144Z"},{"marker":"1_1591211549.499975_770.3","last_update":"2020-06-03T19:12:29.499975Z"},{"marker":"1_1591211562.516577_616.1","last_update":"2020-06-03T19:12:42.516577Z"},{"marker":"1_1591211562.516791_562.1","last_update":"2020-06-03T19:12:42.516791Z"}] 2020-06-03T19:13:25.675 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:13:25.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.676 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:25.718 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:25.718 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:25.975 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1170745615022200934},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.847144_456.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.847144Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211549.499975_770.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:29.499975Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211562.516577_616.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516577Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211562.516791_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516791Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:25.976 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-03T19:13:25.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:25.978 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:26.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:26.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:26.284 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211578.847144_456.1","last_update":"2020-06-03T19:12:58.847144Z"},{"marker":"1_1591211549.499975_770.3","last_update":"2020-06-03T19:12:29.499975Z"},{"marker":"1_1591211562.516577_616.1","last_update":"2020-06-03T19:12:42.516577Z"},{"marker":"1_1591211562.516791_562.1","last_update":"2020-06-03T19:12:42.516791Z"}] 2020-06-03T19:13:26.285 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:13:26.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.286 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:26.614 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3335119464549090040},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.847144_456.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.847144Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211549.499975_770.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:29.499975Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211562.516577_616.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516577Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211562.516791_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516791Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:26.615 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-03T19:13:26.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.616 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:26.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:26.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:26.933 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211578.847144_456.1","last_update":"2020-06-03T19:12:58.847144Z"},{"marker":"1_1591211549.499975_770.3","last_update":"2020-06-03T19:12:29.499975Z"},{"marker":"1_1591211562.516577_616.1","last_update":"2020-06-03T19:12:42.516577Z"},{"marker":"1_1591211562.516791_562.1","last_update":"2020-06-03T19:12:42.516791Z"}] 2020-06-03T19:13:26.933 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:13:26.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:26.935 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:27.251 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11569358039035221345},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.847144_456.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.847144Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211549.499975_770.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:29.499975Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211562.516577_616.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516577Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211562.516791_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:42.516791Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:27.251 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-03T19:13:27.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.253 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:27.333 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:27.341 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:27.595 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591211578.466491_381.2","last_update":"2020-06-03T19:12:58.466491Z"},{"marker":"1_1591211556.446133_652.9","last_update":"2020-06-03T19:12:36.446133Z"},{"marker":"1_1591211578.466491_439.1","last_update":"2020-06-03T19:12:58.466491Z"},{"marker":"1_1591211578.466491_424.1","last_update":"2020-06-03T19:12:58.466491Z"}] 2020-06-03T19:13:27.596 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:13:27.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.598 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:27.641 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:27.641 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:27.890 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4441235937469265296},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_381.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211556.446133_652.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:36.446133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_439.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:27.891 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-03T19:13:27.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:27.892 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:27.933 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:27.934 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:28.168 INFO:teuthology.orchestra.run.smithi061.stdout:[{"marker":"1_1591211578.466491_381.2","last_update":"2020-06-03T19:12:58.466491Z"},{"marker":"1_1591211556.446133_652.9","last_update":"2020-06-03T19:12:36.446133Z"},{"marker":"1_1591211578.466491_439.1","last_update":"2020-06-03T19:12:58.466491Z"},{"marker":"1_1591211578.466491_424.1","last_update":"2020-06-03T19:12:58.466491Z"}] 2020-06-03T19:13:28.168 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:13:28.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.170 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:28.216 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:28.217 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:28.455 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":6026847009502321472},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_381.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211556.446133_652.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:36.446133Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_439.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211578.466491_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.466491Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:28.455 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-03T19:13:28.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.457 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:28.540 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:28.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:28.762 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211578.155289_368.2","last_update":"2020-06-03T19:12:58.155289Z"},{"marker":"1_1591211556.136023_540.9","last_update":"2020-06-03T19:12:36.136023Z"},{"marker":"1_1591211578.155289_336.1","last_update":"2020-06-03T19:12:58.155289Z"},{"marker":"1_1591211578.155289_404.1","last_update":"2020-06-03T19:12:58.155289Z"}] 2020-06-03T19:13:28.762 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:13:28.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:28.764 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:28.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:28.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:29.103 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":17926808424673733064},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_368.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211556.136023_540.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:36.136023Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:29.103 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-03T19:13:29.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:29.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:29.105 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:29.184 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:29.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:29.412 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"1_1591211578.155289_368.2","last_update":"2020-06-03T19:12:58.155289Z"},{"marker":"1_1591211556.136023_540.9","last_update":"2020-06-03T19:12:36.136023Z"},{"marker":"1_1591211578.155289_336.1","last_update":"2020-06-03T19:12:58.155289Z"},{"marker":"1_1591211578.155289_404.1","last_update":"2020-06-03T19:12:58.155289Z"}] 2020-06-03T19:13:29.413 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:13:29.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:29.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:29.414 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:29.458 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:29.708 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":18141103562818399107},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_368.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591211556.136023_540.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:36.136023Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_336.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591211578.155289_404.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:12:58.155289Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-03T19:13:29.709 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-03T19:13:29.710 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-06-03T19:13:29.711 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-65 2020-06-03T19:13:29.755 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=rhmmtn-66 2020-06-03T19:13:29.826 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=rhmmtn-67 2020-06-03T19:13:32.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:32.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:32.260 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:32.304 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:32.305 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:32.561 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.75.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:32.562 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:32.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:32.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:32.563 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:32.610 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:32.610 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:32.861 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:13:32.051155Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:32.862 WARNING:rgw_multi.tests:bucket rhmmtn-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.75.5 target=00000000004.69.5 2020-06-03T19:13:37.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:37.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:37.865 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:38.190 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:13:32.051155Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:38.191 WARNING:rgw_multi.tests:bucket rhmmtn-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.75.5 target=00000000004.69.5 2020-06-03T19:13:43.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:43.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:43.194 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:43.519 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:13:32.051155Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:43.519 WARNING:rgw_multi.tests:bucket rhmmtn-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.75.5 target=00000000004.69.5 2020-06-03T19:13:48.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:48.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:48.522 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:48.575 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:48.575 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:48.837 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.5","timestamp":"2020-06-03T19:13:32.051155Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:48.838 WARNING:rgw_multi.tests:bucket rhmmtn-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.75.5 target=00000000004.69.5 2020-06-03T19:13:53.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:53.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:53.841 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:53.885 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:53.886 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:54.146 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.75.5","timestamp":"2020-06-03T19:13:32.189521Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:54.147 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:54.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.149 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:54.190 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:54.423 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.75.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:54.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:54.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.425 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:54.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:54.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:54.771 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.75.5","timestamp":"2020-06-03T19:13:32.189521Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:54.772 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:54.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:54.773 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:54.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:54.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:55.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.75.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:55.057 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:55.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.059 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:55.104 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:55.104 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:55.385 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.75.5","timestamp":"2020-06-03T19:13:32.189521Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:55.385 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-65 2020-06-03T19:13:55.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.387 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:55.720 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.85.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:55.721 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-65 2020-06-03T19:13:55.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:55.723 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:55.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:55.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:56.039 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:13:52.362419Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:56.039 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-65 2020-06-03T19:13:56.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.042 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:56.093 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:56.094 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:56.351 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.85.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:56.352 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-65 2020-06-03T19:13:56.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.353 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:56.419 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:56.419 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:56.675 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:13:52.366416Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:56.676 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-65 2020-06-03T19:13:56.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:56.677 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:57.007 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.63.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:57.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-65 2020-06-03T19:13:57.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.009 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:57.091 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:57.092 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:57.362 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:13:52.246327Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:57.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-65 2020-06-03T19:13:57.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.364 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:57.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:57.444 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:57.682 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.63.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:13:57.683 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-65 2020-06-03T19:13:57.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:13:57.684 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:13:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:13:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:13:58.005 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-03T19:13:52.250350Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:13:58.006 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-65 2020-06-03T19:13:58.006 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-65 zones={test-zone2, test-zone1} 2020-06-03T19:13:58.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:58.066 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:13:58.067 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:58.077 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:13:58.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:13:58.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:13:58.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:13:59.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-65 zones={test-zone2, test-zone1} 2020-06-03T19:13:59.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-65 zones={test-zone3, test-zone1} 2020-06-03T19:13:59.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:13:59.310 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:13:59.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:13:59.320 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:13:59.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:13:59.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:14:00.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:14:00.346 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-65 zones={test-zone3, test-zone1} 2020-06-03T19:14:00.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-65 zones={test-zone3, test-zone2} 2020-06-03T19:14:00.359 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:00.410 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:14:00.411 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:00.422 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-03T19:14:00.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:14:00.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:14:01.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-03T19:14:01.590 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-65 zones={test-zone3, test-zone2} 2020-06-03T19:14:01.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-06-03T19:14:01.592 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-68 2020-06-03T19:14:01.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:01.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:01.720 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:01.728 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:14:01.724+0000 7fb2be7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.728 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:14:01.724+0000 7fb29d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.730 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:14:01.726+0000 7fcc43fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.731 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:14:01.726+0000 7fcc49ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.732 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:14:01.726+0000 7fc3a6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.732 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:14:01.726+0000 7fc3a4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:01.762 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:01.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:01.926 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:01.954 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211641.629166_360.1","last_update":"2020-06-03T19:14:01.629166Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:01.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:01.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:01.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:01.956 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:01.998 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:01.998 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:02.230 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211641.629166_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:01.629166Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:02.231 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:02.231 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:02.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.233 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:02.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:02.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:02.504 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211641.629166_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:01.629166Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:02.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:02.505 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:02.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.507 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:02.821 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211641.629166_360.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:01.629166Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:02.821 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:02.822 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:02.823 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:02.865 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:02.865 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:03.114 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.35.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.53.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-03T19:14:03.114 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-68 2020-06-03T19:14:03.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:03.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:03.116 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:03.165 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:03.165 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:03.455 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5","timestamp":"2020-06-03T19:14:01.842274Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-03T19:14:01.802218Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-03T19:14:03.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-68 2020-06-03T19:14:03.558 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-06-03T19:14:03.559 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-69 2020-06-03T19:14:03.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:03.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:03.795 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:03.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:03.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:03.995 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:04.031 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211643.592941_367.1","last_update":"2020-06-03T19:14:03.592941Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:04.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:04.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.033 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:04.072 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:04.290 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211643.592941_367.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:03.592941Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:04.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:04.291 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:04.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.293 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:04.334 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:04.335 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:04.539 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211643.592941_367.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:03.592941Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:04.539 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:04.540 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:04.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.542 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:04.583 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:04.584 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:04.861 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211643.592941_367.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:03.592941Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:04.862 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:04.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:04.864 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:04.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:04.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:05.150 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.107.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.65.5"}]} 2020-06-03T19:14:05.150 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:05.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.152 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:05.206 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:05.206 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:05.488 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.107.5","timestamp":"2020-06-03T19:14:03.782553Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-03T19:14:04.018303Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:14:03.782472Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-03T19:14:03.782524Z"}}}] 2020-06-03T19:14:05.489 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:05.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.491 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:05.531 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:05.532 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:05.775 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.107.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.65.5"}]} 2020-06-03T19:14:05.775 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:05.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:05.778 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:05.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:05.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:06.123 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.107.5","timestamp":"2020-06-03T19:14:03.778568Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-03T19:14:04.014303Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:14:03.778490Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-03T19:14:03.778546Z"}}}] 2020-06-03T19:14:06.124 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:06.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.127 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:06.204 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:06.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:06.450 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.107.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.65.5"}]} 2020-06-03T19:14:06.450 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:06.451 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.452 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:06.496 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:06.496 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:06.767 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.107.5","timestamp":"2020-06-03T19:14:03.602287Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-03T19:14:03.752763Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:14:03.706665Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-03T19:14:03.655451Z"}}}] 2020-06-03T19:14:06.768 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-69 2020-06-03T19:14:06.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:06.770 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:06.851 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:06.851 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:07.106 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.85.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.123.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.84.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.91.5"}]} 2020-06-03T19:14:07.107 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-69 2020-06-03T19:14:07.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.120 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:07.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:07.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:07.432 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:14:03.962450Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.123.5","timestamp":"2020-06-03T19:14:04.366344Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5","timestamp":"2020-06-03T19:14:04.166334Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.91.5","timestamp":"2020-06-03T19:14:03.962483Z"}}}] 2020-06-03T19:14:07.433 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-69 2020-06-03T19:14:07.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.434 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:07.476 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:07.476 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:07.722 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.85.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.123.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.84.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.91.5"}]} 2020-06-03T19:14:07.723 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-69 2020-06-03T19:14:07.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:07.724 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:07.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:07.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:08.024 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-03T19:14:03.962380Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.123.5","timestamp":"2020-06-03T19:14:04.366334Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.84.5","timestamp":"2020-06-03T19:14:04.206332Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.91.5","timestamp":"2020-06-03T19:14:03.962409Z"}}}] 2020-06-03T19:14:08.024 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-69 2020-06-03T19:14:08.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.026 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:08.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:08.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:08.340 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.118.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.105.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.72.5"}]} 2020-06-03T19:14:08.340 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-69 2020-06-03T19:14:08.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.342 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:08.423 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:08.423 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:08.658 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.118.5","timestamp":"2020-06-03T19:14:04.054542Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5","timestamp":"2020-06-03T19:14:04.298397Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-06-03T19:14:04.054579Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:14:04.054596Z"}}}] 2020-06-03T19:14:08.658 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-69 2020-06-03T19:14:08.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.660 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:08.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:08.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:08.979 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.118.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.105.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.80.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.72.5"}]} 2020-06-03T19:14:08.980 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-69 2020-06-03T19:14:08.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:08.982 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:09.027 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:09.027 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:09.300 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.118.5","timestamp":"2020-06-03T19:14:04.054470Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5","timestamp":"2020-06-03T19:14:04.294425Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-06-03T19:14:04.054426Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-03T19:14:04.054487Z"}}}] 2020-06-03T19:14:09.301 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-69 2020-06-03T19:14:09.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-69 zones={test-zone2, test-zone1} 2020-06-03T19:14:09.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:09.317 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:09.318 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:09.318 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:09.318 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:09.319 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:09.327 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:09.327 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:09.328 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:09.328 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:09.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:09.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:09.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:09.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:09.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:09.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:09.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:10.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:10.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:10.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:10.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:10.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:10.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-69 zones={test-zone2, test-zone1} 2020-06-03T19:14:10.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-69 zones={test-zone3, test-zone1} 2020-06-03T19:14:10.527 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:10.533 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:10.533 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:10.534 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:10.535 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:10.541 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:10.542 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:10.542 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:10.543 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:10.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:10.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:10.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:10.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:10.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:10.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:11.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:11.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:11.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:11.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:11.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:11.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:11.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-69 zones={test-zone3, test-zone1} 2020-06-03T19:14:11.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-69 zones={test-zone3, test-zone2} 2020-06-03T19:14:11.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:11.699 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:11.700 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:11.700 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:11.701 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:11.701 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:11.720 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:11.721 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:11.721 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:11.723 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:11.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:11.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:11.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:12.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:12.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:12.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:12.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:12.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:12.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:12.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:12.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:12.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:12.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-69 zones={test-zone3, test-zone2} 2020-06-03T19:14:12.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:12.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:12.936 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:14:12.979 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:12.979 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:13.212 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=rhmmtn-70 2020-06-03T19:14:13.333 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:14:13.332+0000 7fb2c4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.334 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2020-06-03T19:14:13.332+0000 7fb2c57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.335 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:14:13.333+0000 7fc3a1ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.336 INFO:tasks.rgw.c2.client.1.smithi061.stdout:2020-06-03T19:14:13.333+0000 7fc3a3ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.337 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:14:13.333+0000 7fcc50ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.337 INFO:tasks.rgw.c2.client.0.smithi061.stdout:2020-06-03T19:14:13.333+0000 7fcc3e7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-03T19:14:13.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.444 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:13.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:13.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:13.648 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:13.687 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211653.248736_371.1","last_update":"2020-06-03T19:14:13.248736Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:13.688 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:13.688 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.689 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:13.737 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:13.738 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:13.947 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:13.947 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:13.948 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:13.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:13.950 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:14.200 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:14.201 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:14.201 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:14.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.203 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:14.256 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:14.257 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:14.536 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:14.536 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:14.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.538 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:14.572 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:14.572 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:14.810 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.69.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.71.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.82.5"}]} 2020-06-03T19:14:14.810 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:14.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:14.812 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:14.856 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:14.856 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:15.123 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:14:13.462293Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:14:13.582264Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5","timestamp":"2020-06-03T19:14:13.622262Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-03T19:14:13.423512Z"}}}] 2020-06-03T19:14:15.124 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:15.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.126 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:15.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:15.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:15.404 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.69.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.71.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.82.5"}]} 2020-06-03T19:14:15.405 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:15.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.407 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:15.492 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:15.493 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:15.752 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:14:13.446313Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:14:13.626304Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5","timestamp":"2020-06-03T19:14:13.626285Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-03T19:14:13.450338Z"}}}] 2020-06-03T19:14:15.752 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:15.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:15.754 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:15.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:15.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:16.073 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.69.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.71.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.82.5"}]} 2020-06-03T19:14:16.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:16.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.075 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:16.396 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-03T19:14:13.259130Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:14:13.402463Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.71.5","timestamp":"2020-06-03T19:14:13.354891Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-03T19:14:13.307037Z"}}}] 2020-06-03T19:14:16.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-70 2020-06-03T19:14:16.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.399 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:16.478 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:16.478 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:16.727 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.99.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.101.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.90.5"}]} 2020-06-03T19:14:16.728 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-70 2020-06-03T19:14:16.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:16.730 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:16.774 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:16.774 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:17.033 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-03T19:14:13.770425Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:14:13.770396Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-03T19:14:14.010338Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5","timestamp":"2020-06-03T19:14:13.730467Z"}}}] 2020-06-03T19:14:17.034 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=rhmmtn-70 2020-06-03T19:14:17.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.036 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:17.078 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:17.079 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:17.340 INFO:teuthology.orchestra.run.smithi061.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.99.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.101.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.90.5"}]} 2020-06-03T19:14:17.341 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-70 2020-06-03T19:14:17.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.344 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:17.389 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:17.389 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:17.622 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-03T19:14:13.810391Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-03T19:14:13.810425Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-03T19:14:13.970374Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5","timestamp":"2020-06-03T19:14:13.702361Z"}}}] 2020-06-03T19:14:17.623 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=rhmmtn-70 2020-06-03T19:14:17.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.624 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:17.906 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.80.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.57.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.82.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.74.5"}]} 2020-06-03T19:14:17.907 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-70 2020-06-03T19:14:17.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:17.909 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:17.995 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:17.995 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:18.260 INFO:teuthology.orchestra.run.smithi112.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-06-03T19:14:13.654347Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:14:13.906316Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-03T19:14:13.898322Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:14:13.654321Z"}}}] 2020-06-03T19:14:18.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=rhmmtn-70 2020-06-03T19:14:18.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:18.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:18.263 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:18.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:18.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:18.582 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.80.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.57.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.82.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.74.5"}]} 2020-06-03T19:14:18.582 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-70 2020-06-03T19:14:18.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:18.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'rhmmtn-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:18.584 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket rhmmtn-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:18.630 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:18.630 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:18.906 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5","timestamp":"2020-06-03T19:14:13.654372Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-03T19:14:13.858252Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-03T19:14:13.830277Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-03T19:14:13.654423Z"}}}] 2020-06-03T19:14:18.907 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=rhmmtn-70 2020-06-03T19:14:18.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-70 zones={test-zone2, test-zone1} 2020-06-03T19:14:18.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:18.925 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:18.925 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:18.926 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:18.926 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:18.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:18.936 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:18.937 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:18.937 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:18.938 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:18.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:19.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:19.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:19.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:19.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:19.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:19.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:19.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:19.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:19.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:19.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:19.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:20.099 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-70 zones={test-zone2, test-zone1} 2020-06-03T19:14:20.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-70 zones={test-zone3, test-zone1} 2020-06-03T19:14:20.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:20.117 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:20.118 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:20.118 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:20.119 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:20.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:20.125 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:20.126 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:20.126 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:20.127 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:20.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:20.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:20.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:20.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:20.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:20.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:20.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:20.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:20.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:20.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:21.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:21.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:21.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-70 zones={test-zone3, test-zone1} 2020-06-03T19:14:21.251 INFO:tasks.rgw_multi.zone_rados:comparing bucket=rhmmtn-70 zones={test-zone3, test-zone2} 2020-06-03T19:14:21.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-03T19:14:21.268 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:21.269 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:21.269 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:21.270 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:21.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-03T19:14:21.278 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-03T19:14:21.278 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-03T19:14:21.279 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-03T19:14:21.279 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-03T19:14:21.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:21.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:21.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-03T19:14:21.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:21.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:21.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-03T19:14:21.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:21.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:22.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-03T19:14:22.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:22.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:22.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-03T19:14:22.447 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=rhmmtn-70 zones={test-zone3, test-zone2} 2020-06-03T19:14:22.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:22.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:22.450 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:14:22.490 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:22.490 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:22.755 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:22.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'rhmmtn-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:22.757 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket rhmmtn-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:22.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:22.840 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:23.082 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:14:23.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:23.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:23.084 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:23.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:23.401 INFO:teuthology.orchestra.run.smithi112.stdout:[{"op_id":"1#00000000001.107.5","op_tag":"12983a69-4226-40f7-aaec-4a03acd75b02.7829.4626","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-03T19:14:03.602287169Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-69:12983a69-4226-40f7-aaec-4a03acd75b02.7833.20"}]},{"op_id":"5#00000000001.99.5","op_tag":"12983a69-4226-40f7-aaec-4a03acd75b02.7829.4650","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-03T19:14:03.752763859Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-69:12983a69-4226-40f7-aaec-4a03acd75b02.7833.20"}]},{"op_id":"8#00000000001.76.5","op_tag":"12983a69-4226-40f7-aaec-4a03acd75b02.7829.4630","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-03T19:14:03.706665205Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-69:12983a69-4226-40f7-aaec-4a03acd75b02.7833.20"}]},{"op_id":"10#00000000001.65.5","op_tag":"12983a69-4226-40f7-aaec-4a03acd75b02.7829.4628","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-03T19:14:03.655451717Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"12983a69-4226-40f7-aaec-4a03acd75b02:rhmmtn-69:12983a69-4226-40f7-aaec-4a03acd75b02.7833.20"}]}] 2020-06-03T19:14:23.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:23.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-03T19:14:23.403 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-03T19:14:23.482 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:23.483 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:24.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:24.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'rhmmtn-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:24.998 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket rhmmtn-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:25.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:25.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:25.334 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2020-06-03T19:14:25.335 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-06-03T19:14:25.381 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-06-03T19:14:25.382 INFO:tasks.rgw_multisite_tests: 2020-06-03T19:14:25.383 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-06-03T19:14:25.384 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1421.531s 2020-06-03T19:14:25.385 INFO:tasks.rgw_multisite_tests: 2020-06-03T19:14:25.385 INFO:tasks.rgw_multisite_tests:OK 2020-06-03T19:14:25.389 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-06-03T19:14:25.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.393 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:25.651 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:25.688 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211653.248736_371.1","last_update":"2020-06-03T19:14:13.248736Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:25.688 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:25.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.690 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:25.724 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:25.725 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:25.977 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:25.977 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:25.978 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:25.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:25.980 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:26.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:26.257 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:26.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:26.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:26.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.260 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:26.301 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:26.301 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:26.582 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:26.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:26.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.584 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:26.624 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:26.624 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:26.827 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:26.865 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211653.248736_371.1","last_update":"2020-06-03T19:14:13.248736Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:26.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:26.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:26.965 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:27.018 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:27.018 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:27.288 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211653.248736_371.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:13.248736Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:27.288 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:27.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:27.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:27.346 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:27.384 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:27.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:27.624 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211667.338726_376.1","last_update":"2020-06-03T19:14:27.338726Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:27.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:27.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:27.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:27.627 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:27.669 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:27.670 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:27.935 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211667.338726_376.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:27.338726Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:27.936 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:28.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.050 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:28.089 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:28.090 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:28.295 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:28.330 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211667.338726_376.1","last_update":"2020-06-03T19:14:27.338726Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:28.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:28.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.332 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:28.374 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:28.375 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:28.653 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211667.338726_376.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:27.338726Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:28.654 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:28.763 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-06-03T19:14:28.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:28.765 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:28.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:28.803 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:28.999 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:29.039 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211668.757584_378.1","last_update":"2020-06-03T19:14:28.757584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:29.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:29.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.042 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:29.083 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:29.083 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:29.313 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211668.757584_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:28.757584Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:29.313 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:29.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:29.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.316 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:29.355 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:29.355 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:29.595 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211668.757584_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:28.757584Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:29.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:29.596 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:29.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.598 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:29.639 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:29.639 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:29.912 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211668.757584_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:28.757584Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:29.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:29.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:29.914 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:29.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:29.954 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:30.152 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:30.188 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211668.757584_378.1","last_update":"2020-06-03T19:14:28.757584Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:30.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:30.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.190 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:30.231 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:30.231 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:30.503 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211668.757584_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:28.757584Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:30.504 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:30.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.566 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:30.603 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:30.799 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:30.831 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211670.560520_382.1","last_update":"2020-06-03T19:14:30.560520Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:30.832 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:30.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:30.834 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:30.889 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:30.889 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:31.161 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211670.560520_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:30.560520Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:31.162 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:31.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:31.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:31.239 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:31.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:31.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:31.524 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211670.560520_382.1","last_update":"2020-06-03T19:14:30.560520Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:31.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:31.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:31.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:31.526 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:31.568 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:31.569 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:31.842 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211670.560520_382.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:30.560520Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:31.842 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:32.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:32.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'rhmmtn-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:32.276 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket rhmmtn-72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:32.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:32.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:32.552 INFO:teuthology.orchestra.run.smithi112.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.92.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.79.5"},{"key":4,"val":"00000000001.76.5"},{"key":5,"val":"00000000001.77.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.111.5"},{"key":8,"val":"00000000001.116.5"},{"key":9,"val":"00000000001.106.5"},{"key":10,"val":"00000000002.73.5"}]} 2020-06-03T19:14:32.552 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-72 2020-06-03T19:14:32.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:32.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'rhmmtn-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:32.554 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket rhmmtn-72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:32.598 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:32.598 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:32.857 INFO:teuthology.orchestra.run.smithi061.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.92.5","timestamp":"2020-06-03T19:14:32.189723Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-03T19:14:31.850481Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-03T19:14:32.145675Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5","timestamp":"2020-06-03T19:14:31.969829Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.111.5","timestamp":"2020-06-03T19:14:31.923476Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.116.5","timestamp":"2020-06-03T19:14:32.101696Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.106.5","timestamp":"2020-06-03T19:14:31.878191Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.5","timestamp":"2020-06-03T19:14:32.233131Z"}}}] 2020-06-03T19:14:32.858 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=rhmmtn-72 2020-06-03T19:14:32.875 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:31.878191Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '1', 'size': 0, 'etag': 'e22f67f0', 'versionId': '', 'sequencer': '98F6D75E6B810302', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.033784.e22f67f0', 'opaqueData': ''} 2020-06-03T19:14:32.876 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:31.923476Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '2', 'size': 0, 'etag': '2ec17130', 'versionId': '', 'sequencer': '98F6D75EF68EFA02', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.049975.2ec17130', 'opaqueData': ''} 2020-06-03T19:14:32.877 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:31.850481Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '0', 'size': 0, 'etag': '753fb1f5', 'versionId': '', 'sequencer': '98F6D75ECAD96505', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.090560.753fb1f5', 'opaqueData': ''} 2020-06-03T19:14:32.877 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:31.969829Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '3', 'size': 0, 'etag': '4cbab075', 'versionId': '', 'sequencer': '98F6D75E753A6D0D', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.225262.4cbab075', 'opaqueData': ''} 2020-06-03T19:14:32.878 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.013846Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '4', 'size': 0, 'etag': '97fabea9', 'versionId': '', 'sequencer': '98F6D75EDC60700D', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.225468.97fabea9', 'opaqueData': ''} 2020-06-03T19:14:32.878 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.189723Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '8', 'size': 0, 'etag': 'c820fd38', 'versionId': '', 'sequencer': '98F6D75E29B13C10', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.272412.c820fd38', 'opaqueData': ''} 2020-06-03T19:14:32.879 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.057857Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '5', 'size': 0, 'etag': '34743afd', 'versionId': '', 'sequencer': '98F6D75EF91C9211', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.294788.34743afd', 'opaqueData': ''} 2020-06-03T19:14:32.880 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.101696Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '6', 'size': 0, 'etag': 'cca4a8ac', 'versionId': '', 'sequencer': '98F6D75EB90E9611', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.295046.cca4a8ac', 'opaqueData': ''} 2020-06-03T19:14:32.880 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.233131Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '9', 'size': 0, 'etag': 'c5117dcd', 'versionId': '', 'sequencer': '98F6D75E2E442D14', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.338510.c5117dcd', 'opaqueData': ''} 2020-06-03T19:14:32.881 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-03T19:14:32.145675Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'rhmmtn-72_notif', 'bucket': {'name': 'rhmmtn-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::rhmmtn-72', 'id': '12983a69-4226-40f7-aaec-4a03acd75b02.7833.23'}, 'object': {'key': '7', 'size': 0, 'etag': 'de83be61', 'versionId': '', 'sequencer': '98F6D75EB32C6E1D', 'metadata': [], 'tags': []}}, 'eventId': '1591211672.493759.de83be61', 'opaqueData': ''} 2020-06-03T19:14:32.893 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-06-03T19:14:33.083 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-06-03T19:14:33.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.085 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:33.121 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:33.121 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:33.311 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:33.348 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211673.078823_386.1","last_update":"2020-06-03T19:14:33.078823Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:33.349 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:33.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.351 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:33.393 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:33.393 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:33.659 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211673.078823_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:33.078823Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:33.659 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:33.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:33.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.662 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:33.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:33.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:33.939 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211673.078823_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:33.078823Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:33.939 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:33.940 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:33.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:33.941 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:33.982 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:34.253 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211673.078823_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:33.078823Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:34.254 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:34.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.256 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:34.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:34.294 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:34.492 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:34.528 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211673.078823_386.1","last_update":"2020-06-03T19:14:33.078823Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:34.529 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:34.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.531 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:34.572 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:34.572 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:34.850 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211673.078823_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:33.078823Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:34.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:34.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:34.902 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:34.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:35.128 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:35.162 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211674.896780_390.1","last_update":"2020-06-03T19:14:34.896780Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:35.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:35.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.164 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:35.205 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:35.206 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:35.476 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211674.896780_390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:34.896780Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:35.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:35.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.558 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:35.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:35.797 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:35.835 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211674.896780_390.1","last_update":"2020-06-03T19:14:34.896780Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:35.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:35.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:35.839 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:35.880 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:35.880 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:36.159 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211674.896780_390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:34.896780Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:36.159 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:36.278 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2020-06-03T19:14:36.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.280 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:36.317 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:36.317 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:36.517 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:36.554 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:36.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:36.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.556 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:36.598 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:36.599 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:36.843 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:36.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:36.845 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:36.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:36.847 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:36.886 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:36.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:37.125 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:37.126 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:37.126 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:37.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.129 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:37.170 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:37.171 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:37.542 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:37.542 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:37.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.545 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:37.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:37.588 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:37.802 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:37.840 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:37.841 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:37.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:37.843 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:37.885 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:37.886 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:38.158 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:38.159 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:38.293 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2020-06-03T19:14:38.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.296 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:38.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:38.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:38.549 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:38.581 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:38.582 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:38.583 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.584 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:38.626 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:38.627 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:38.884 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:38.885 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:38.886 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:38.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:38.888 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:38.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:38.930 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:39.171 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:39.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:39.172 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:39.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.174 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:39.218 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:39.493 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:39.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:39.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.495 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:39.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:39.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:39.734 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:39.769 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:39.770 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:39.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:39.772 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:39.815 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:39.815 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:40.080 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:40.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:40.082 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2020-06-03T19:14:40.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.083 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:40.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:40.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:40.379 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:40.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-03T19:14:40.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.381 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:40.426 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:40.426 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:40.690 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:40.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-03T19:14:40.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-03T19:14:40.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.694 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:40.981 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:40.982 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-03T19:14:40.982 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:40.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:40.984 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:41.024 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:41.025 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:41.298 INFO:teuthology.orchestra.run.smithi061.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"b2e295e4-c2ab-4a12-b669-6be057e86c4f","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591210282.699710_124.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-03T18:51:22.699710Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591211676.273709_392.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-03T19:14:36.273709Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-03T19:14:41.299 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-03T19:14:41.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:41.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:41.301 INFO:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:41.342 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:41.342 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2020-06-03T19:14:41.560 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b2e295e4-c2ab-4a12-b669-6be057e86c4f 2020-06-03T19:14:41.597 INFO:teuthology.orchestra.run.smithi112.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1591211676.273709_392.1","last_update":"2020-06-03T19:14:36.273709Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-03T19:14:41.597 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-03T19:14:41.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:41.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-03T19:14:41.599 INFO:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-03T19:14:41.640 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setuser ceph since I am not root 2020-06-03T19:14:41.640 INFO:teuthology.orchestra.run.smithi061.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T06:31:20.224 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-06-04T06:31:20.253 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2020-06-04T06:31:20.257 DEBUG:teuthology.task.console_log:Killing console logger for smithi061 2020-06-04T06:31:20.257 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2020-06-04T06:31:20.258 DEBUG:teuthology.task.console_log:Killing console logger for smithi061 2020-06-04T06:31:20.258 DEBUG:teuthology.exit:Finished running handlers